System, method, architecture, and computer program product for dynamic power management in a computer system

ABSTRACT

Network architecture, computer system and/or server, circuit, device, apparatus, method, and computer program and control mechanism for managing power consumption and workload in computer system and data and information servers. Further provides power and energy consumption and workload management and control systems and architectures for high-density and modular multi-server computer systems that maintain performance while conserving energy and method for power management and workload management. Dynamic server power management and optional dynamic workload management for multi-server environments is provided by aspects of the invention. Modular network devices and integrated server system, including modular servers, management units, switches and switching fabrics, modular power supplies and modular fans and a special backplane architecture are provided as well as dynamically reconfigurable multi-purpose modules and servers. Backplane architecture, structure, and method that has no active components and separate power supply lines and protection to provide high reliability in server environment.

RELATED APPLICATIONS

This application is a continuation of and claims the benefit of priorityunder 35 U.S.C. §120 to U.S. Utility patent application Ser. No.09/860,373 filed 18 May 2001 and now U.S. Pat. No. ______; which claimsthe benefit of priority under 35 U.S.C. §119(e) to U.S. ProvisionalApplication Ser. No. 60/283,375 entitled System, Method And ArchitectureFor Dynamic Server Power Management And Dynamic Workload Management forMulti-Server Environment filed 11 Apr. 2001; U.S. ProvisionalApplication Ser. No. 60/236,043 entitled System, Apparatus, and Methodfor Power-Conserving Multi-Node Server Architecture filed 27 Sep. 2000;and U.S. Provisional Application Ser. No. 60/236,062 entitled System,Apparatus, and Method for Power Conserving and Disc-Drive LifeProlonging RAID Configuration filed 27 Sep. 2000; each of which patentand patent application is hereby incorporated by reference.

FIELD OF THE INVENTION

This invention pertains generally to architecture, apparatus, systems,methods, and computer programs and control mechanisms for managing powerconsumption and work-load in data and information servers; moreparticularly to power consumption and workload management and controlsystems for high-density multi-server computer system architectures thatmaintain performance while conserving energy and to the method for powermanagement and workload management used therein, and most particularlyto system, method, architectures, and computer programs for dynamicserver power management and dynamic workload management for multi-serverenvironments.

BACKGROUND

Heretofore, servers generally, and multi-node network servers inparticular, have paid little if any attention to power or energyconservation. Such servers were designed and constructed to run at ornear maximum levels so as to serve data or other content as fast aspossible, or where service demands were less than capacity to remainever vigilant to provide fast response to service requests. Increasingprocessor and memory speeds have typically been accompanied by higherprocessor core voltages to support the faster device switching times,and faster hard disk drives have typically lead to faster and moreenergy-hungry disk drive motors. Larger memories and caches have alsolead to increased power consumption even for small single-node servers.Power conservation efforts have historically focused on the portablebattery-powered notebook market where battery life is an importantmarketing and use characteristic. However, in the server area, littleattention has been given to saving power, such servers usually notadopting or utilizing even the power conserving suspend, sleep, orhibernation states that are available with some Microsoft 95/98/2000,Linux, Unix, or other operating system based computers, personalcomputers, PDAs, or information appliances.

Multi-node servers present a particular energy consumption problem asthey have conventionally be architected as a collection of large powerhungry boxes interconnected by external interconnect cables. Littleattention has been placed on the size or form factor of such networkarchitectures, the expansibility of such networks, or on the problemsassociated with large network configurations. Such conventional networkshave also by-and-large paid little attention to the large amounts ofelectrical power consumed by such configurations or in the savingspossible. This has been due in part because of the rapid and unexpectedexpansion in the Internet and in servers connected with and serving toInternet clients. Internet service companies and entrepreneurs have beenmore interested in a short time to market and profit than on the effecton electrical power consumption and electrical power utilities; however,continuing design and operation without due regard to power consumptionin this manner is problematic.

Networks servers have also by-and-large neglected to factor into theeconomics of running a network server system the physical plant costassociated with large rack mounted equipment carrying perhaps onenetwork node per chassis. These physical plant and real estate costsalso contribute to large operating costs.

In the past, more attention was given to the purchase price of equipmentand little, attention to the operating costs. It would be apparent tothose making the calculation that operating costs may far exceed initialequipment purchase price, yet little attention has been paid to thisfact. More recently, the power available in the California electricalmarket has been at crisis levels with available power reserves droppingbelow a few percent reserve and rolling blackouts occurring aselectrical power requirements drop below available electrical powergeneration capacity. High technology companies in the heart of SiliconValley cannot get enough electrical power to make or operate product,and server farms which consume vast quantities of electrical energy forthe servers and for cooling equipment and facilities in which they arehoused, have stated that they may relocated to areas with stablesupplies of low-cost electricity.

Even were server manufactures motivated to adopt available powermanagement techniques, such techniques represent only a partialsolution. Conventional computer system power management tends to focuson power managing a single CPU, such as by monitoring certain restrictedaspects of the single CPU operation and making a decision that the CPUshould be run faster to provide greater performance or more slowly toreduce power consumption.

Heretofore, computer systems generally, and server systems having aplurality of servers where each server includes at least one processoror central processing unit (CPU) in particular have not been powermanaged to maintain performance and reduce power consumption. Even wherea server system having more than one server component and CPU maypossibly have utilized a conventional personal computer architecturethat provided some measure of localized power management separatelywithin each CPU, no global power management architecture or methods haveconventionally been applied to power manage the set of servers and CPUsas a single entity.

The common practice of over-provisioning a server system so as to beable to meet peak demands has meant that during long periods of time,individual servers are consuming power and yet doing no useful work, orseveral servers are performing some tasks that could be preformed by asingle server at a fraction of the power consumption.

Operating a plurality of servers, including their CPU, hard disk drive,power supply, cooling fans, and any other circuits or peripherals thatare associated with the server, at such minimal loading alsounnecessarily shortens their service life. However, conventional serversystems do not consider the longevity of their components. To the extentthat certain of the CPUs, hard disk drives, power supplies, and coolingfans may be operated at lower power levels or for mechanical systems(hard disk drive and cooling fans in particular) their effective servicelife may be extended.

Therefore there remains a need for a network architecture and networkoperating method that provides large capacity and multiple network nodesor servers in a small physical footprint and that is power conservativerelative to server performance and power consumed by the server, as wellas power conservative from the standpoint of power for server facilityair conditioning. These and other problems are solved by the inventivesystem, apparatus and method. There also remains a need for server farmsthat are power managed in an organized global manner so that performanceis maintained while reducing power consumption. There also remains aneed to extend the effective lifetime of computer system components andservers so that the total cost of ownership is reduced.

SUMMARY

Aspects of the invention provide network architecture, computer systemand/or server, circuit, device, apparatus, method, and computer programand control mechanism for managing power consumption and workload incomputer system and data and information servers. Further provides powerand energy consumption and workload management and control systems andarchitectures for high-density and modular multi-server computer systemsthat maintain performance while conserving energy and method for powermanagement and workload management. Dynamic server power management andoptional dynamic workload management for multi-server environments isprovided by aspects of the invention. Modular network devices andintegrated server system, including modular servers, management units,switches and switching fabrics, modular power supplies and modular fansand a special backplane architecture are provided as well as dynamicallyreconfigurable multi-purpose modules and servers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic illustration showing a exemplary embodiment ofan inventive power conserving high-density server system.

FIG. 2 is a diagrammatic illustration showing an exemplary embodiment ofa single 2U high rack mountable Integrated Server System Unit having aplurality of modular server units.

FIG. 3 is a diagrammatic illustration showing a standard server farmarchitecture in which multiple nodes are individually connected bycables to each other to form the desired network.

FIG. 4 is a diagrammatic illustration showing an embodiment of theinventive Integrated Appliance Server (IAS) standard architecture alsoor alternatively referred to as an Integrated Server System (ISS)architecture in which multiple nodes selected from at least a computernode (CN) such as a server module (SM), network node (NN) also referredto as a switch module, and monitor or management node (MN) also referredto as a Management Module (MM) are provided within a common enclosureand coupled together via an internal backplane bus.

FIG. 5 is a diagrammatic illustration showing another embodiment of theinvention in which multiple modular IAS (or ISS) clusters eachcontaining multiple nodes are cascaded to define a specialized system.

FIG. 6 is a diagrammatic illustration showing an embodiment of anIntegrated Server System Architecture having two interconnectedintegrated server system units (ISSUs) and their connectivity with theexternal world.

FIG. 7 is a diagrammatic illustration showing an exemplary embodiment ofan AMPC bus and the connectivity of Server Modules and ManagementModules to the bus to support serial data, video, keyboard, mouse, andother communication among and between the modules.

FIG. 8 is a diagrammatic illustration showing an exemplary embodiment ofISSU connectivity to gigabit switches, routers, load balances, and anetwork.

FIG. 9 is a diagrammatic illustration showing an embodiment of theinventive power conserving power management between two servers and amanager.

FIG. 10 is a diagrammatic illustration showing an alternative embodimentof a server system showing detail as to how activity may be detected andoperating mode and power consumption controlled in response.

FIG. 11 is a diagrammatic illustration showing another alternativeembodiment of a server system particular adapted for a Transmeta Crusoe™type processor having LongRun™ features showing detail as to howactivity may be detected and operating mode and power consumptioncontrolled in response.

FIG. 12 is a diagrammatic illustration showing aspects of theconnectivity of two management modules to a plurality of server modulesand two Ethernet switch modules.

FIG. 13 is a diagrammatic illustration showing an exemplary internetworkand the manner in which two different types of master may be deployed topower manage such system.

FIG. 14 is a diagrammatic illustration showing a graph of the CPUutilization (processor activity) as a function of time, wherein the CPUutilization is altered by entering different operating modes.

FIG. 15 is a diagrammatic illustration showing an exemplary state enginestate diagram graphically illustrating the relationships amongst themodes and identifying some of the transitions between states or modesfor operation of an embodiment of the inventive system and method.

FIGS. 16-23 are diagrammatic illustrations showing exemplary statediagram for operating mode transitions.

FIG. 24 is a diagrammatic illustration showing the manner in which aplurality of servers may operate in different modes based on localdetection and control of selected mode transitions and local detectionbut global control of other selected mode transitions.

FIG. 25 is a diagrammatic illustration showing an embodiment of acomputer system having a plurality of hard disc drives configured in aRAID configuration and using a separate RAID hardware controller.

FIG. 26 is a diagrammatic illustration showing an alternative embodimentof a computer system having a plurality of hard disc drives configuredin a RAID configuration and using software RAID control in the hostprocessor.

FIG. 27 is a diagrammatic illustration showing an exemplary RAID 1configuration.

FIG. 28 is a diagrammatic illustration showing an exemplary RAID 0+1(RAID 10) configuration.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The present invention pertains to computer system architectures andstructures and methods for operating such computer system architecturesin a compact high-performance low-power consumption manner. Computers,information appliances, data processing systems, and all manner ofelectronic systems and devices may utilize and benefit from theinnovations described herein. Aspects of the invention also contributeto reliability, ease of maintenance, and longevity of the system as awhole and operation components thereof. In an application that is ofparticular importance and which benefits greatly from the innovationsdescribed here, the computer system is or includes a server systemhaving at least one and more typically a plurality of servers. Eachserver will include at least one processor or CPU but may includemultiple CPUs. In multiple server configurations significant powerconsumption reduction is achieved by applying the inventive powermanagement scheme. These and other aspects of the invention aredescribed in the sections that follow.

The physical form factors of the server modules and management modulesprovide significant advantages, however, it will be appreciated that theinvention need not be limited to such modular servers or modularmanagement elements, and that the invention extends to discrete serversand management elements. It is also to be appreciated that although theexemplary embodiments focus attention toward servers, server systems,and power saving features for server systems, that aspects of theinvention transcend such servers and server environments. For example,distributed computer systems of all types may benefit from the form ofcoordinated management and control to determine CPU loading andcoordinate computational processing over a multiplicity of processors.

Section headers, where provided, are merely for the convenience of thereader and are not to be taken as limiting the scope of the invention inany way, as it will be understood that certain elements and features ofthe invention have more than one function and that aspects of theinvention and particular elements are described throughout thespecification.

With respect to FIG. 1 there is shown an exemplary rack mounted serversystem 50. The rack carries a plurality of 2U high integrated serversystem units 52 each having one or more management modules (MM) 53 andone or more server modules (SM) 54, each server module providing a fullyindependent server. Each server includes a processor or CPU and memory,mass storage device such as a hard disk drive, and input/output ports.In the embodiment illustrated each 2U high chassis 55 has 16 slots eachof which may contain a PC-board mounted server module 54 or managementmodule 53. The chassis 55 also provides one or more power supplies 56and one or more cooling fan banks 57. These elements are coupled forcommunication by switches 59 and a backplane 58.

The different ISS chassis units 55 may be coupled together to form alarger system and these server units share a gigabit uplink 60, loadbalancer 61, a router 62 to connect to a network such as the Internet63. Network Attached Storage (NAS) 64 may desirably be provided toincrease storage capacity over that provided in individual servermodules. Local and/or remote management nodes or workstations 65 may beprovided to permit access to the system 50. As power management is animportant feature of aspects of the invention, the provision of electricservice 66 to the system 50 as well as electric service 68 to buildingor facilities air conditioning or cooling 69 is also illustrated.Content or data may readily be served to remote clients 70 over theInternet 63.

The illustration in FIG. 1 shows how the form factor of the server andmanagement modules increases server density and reduces the footprint ofthe server system. Of course multiple racks may be added to increasesystem capacity. The inventive power management feature extends toindividual server modules, to groups of server modules, and to theentire set of server modules in the system 50 as desired. Powermanagement may also be applied to the management modules, power supplymodules, switches, cooling fan modules, and other components of the ISS.

An exemplary embodiment of an ISS unit is illustrated in FIG. 2, whichshows the manner in which PC board based server modules and managementmodules plug into a back plane along with power supplies, cooling fanunits, switches, and other components to provide the high-densitysystem. These and other features are described in greater detail in theremainder of this specification.

With respect to FIG. 3, there is shown in diagrammatic form, anillustration showing a standard server farm architecture in whichmultiple nodes are individually connected by cables to each other toform the desired network. Server farms such as this are typically powerhungry, operate continuously with little or no regard for actual usage,have a large footprint, and generate large amounts of heat that requireconsiderable air conditioning to dissipate or remove.

FIG. 4 is a diagrammatic illustration showing an embodiment of theinventive Integrated Server System (ISS) standard architecture in whichmultiple nodes selected from at least a computer node (CN) or ServerModule (SM), network node (NN) or Switch Module (SWM), and monitor node(MN) or Management Module (MM) are provided within a common enclosureand coupled together via an internal backplane bus and internal switch.Two separate switching fabrics sw1 and sw0 are provided and describedhereinafter. Up-link (up0 and up1) and down-link (down0 and down1) areprovided to permit cascading multiple ISS cluster units. Monitor nodes(MN or MonX) such as Mon0 and Mon1 are coupled or connected via any oneor more of serial I/O interfaces, RJ-45 interfaces, and RJ-11 modeminterfaces to each switching node or other switching means, network node(NN), or to a network node via a switching node or other means.

FIG. 5 is a diagrammatic illustration showing another embodiment of theinvention in which multiple modular ISS clusters each containingmultiple nodes are cascaded to define a specialized system. This is anexample of the manner in which multiple nodes within an ISS unit andmultiple cascaded ISS units may be transformed or morphed to suitnetwork configuration requirements.

It is noted that each Integrated Appliance Server (IAS) or IntegratedServer System (ISS) cluster desirably includes some intelligence. Inorder to configure there is some master that is selected duringinitialization of the system, such as when it is booted or reset. Thesystem can be designed such that any one of the nodes can be the masternode. For example, one node may be designated as the master or the firstnode that becomes available after initialization, boot, or reset mayassume the role of master node. There is no need for a separateprocessor or control within the box or enclosure. The master can controlthe rest of the system. Factors used in such control include the load,the quality of service desired or required. The system can reconfigureitself at any time in real-time in response to conditions encounteredand predetermined or adaptive rules or procedures. For example, induring a period of time the number of email requests increases and thenumber of web page requests decreases or is static, then nodes mayconverted to serve email so that the email service capacity andperformance are increased to handle the additional load. A node can alsoserve more than one function, for example it can function to serve emailand web pages and can be self balancing.

The architecture or topology may be morphed or transformed into manyalternative structures. All nodes are connected by an internal backplanethereby eliminate the need for external and fragile connectors andcables. Each node can be adapted to perform any one of numerousfunctions, or a plurality of the functions concurrently. Any node can bea cache node, an email node, a web page server node, or the like.Selection of the function or functions of the nodes are selected(manually or automatically) based on such factors as the load for eachtype of function and the desired level or quality of service (QOS) forthat function. For example, if rapid web page service is desired ascompared to email service, more node resources may be allocated toserving web pages.

All nodes are reconfigurable at any time based on circumstances, such asload and QOS. For example, if only need to serve so many pages persecond then may choose not to allocate additional node resources to webpage serving. In some instances, the tasks performed by one node (suchas node serving web pages) may be shifted to one or more other nodesthat have additional capacity, and that former web server node powereddown or put into another power or energy saving mode. This adaptivereconfiguration and distribution of node functions maintains QOS whileminimizing power consumption, heat dissipation, and other negative ordetrimental effects. Placing the equipment or portions of the equipmentin to power saving modes or standby modes also has the potential benefitof prolonging effective service life.

The power consumption of each node is therefore also adjustable based onthe load and/or QOS requirement. On one level this adjustment is enabledby using or not using one or more nodes, and at a second level, theperformance characteristics of the node may be adapted or configured tosuit operational requirements. For example, a processor clock speed maybe increased when demands are high and decreased or turned off whendemands are modest or there is no demand. Again, these adjustments maybe made automatically based on sensed load and feedback as to whetherquality of service requirements have been met.

The invention also provides a functional and architectural topology inwhich each node represents a cell in a network of interconnected cells.These nodes or cells are linked and interoperate with each other suchthat when the operating characteristics of one node change in responseto a command or sensed conditions (e.g. current loading and/or QOS) theother nodes become aware of this change and may also optionally butdesirably be reflected in reconfiguration of other of the nodes.Advantageously, the number or frequency of such changes may becontrolled so that the system remains stable. For example,reconfiguration may be limited in frequency or predetermined delays maybe built into the system so that a settling time is provided after eachnode is reconfigured.

Other intelligence can be put into the node clusters if desired. Recallthat a cluster includes a set of interconnected nodes, in a preferredembodiment each cluster includes 16 nodes in a single physicalenclosure.

Each ISS consists of multiple nodes. Nodes may be configured as computernodes, monitor nodes, network nodes, and any other type of node known inthe art. Normally, the nodes are physically housed in a single box orenclosure and connected by an enclosure backplane. The architecture maybe morphed or transformed into many different alternative organizations.For example, the ISS standard architecture may be configures into aserver farm. This can be done for either the entire ISS, a part of asingle ISS, or among multiple ISS units.

The computer nodes (also known as server nodes or server modules) may beconfigured or mapped to email, FTP, or Web nodes. One or more of suchcomputer nodes may then be coupled together with other nodes. Thisexemplary first implementation is illustrated as the inner box in FIG.5. Each node may be configured in any way desired as in at least oneembodiment of the invention, the structure and function of each node atthe time of manufacture is identical, and any one of such nodes may beplaced in service or later reconfigured to provide the desiredfunctionality. In one embodiment, each computer node type is the samewhile in other embodiments they are of different types.

Furthermore, in one embodiment, every node in a cluster of nodes isidentical as they come from the factory, and any node may be adapted,such as through software that is loaded into a node, to provide any oneof a plurality of available functions. In another embodiment, somewhatto very different node structures are provided within a single clusterto provide more highly optimized network nodes, computer nodes, andmonitor nodes. The existence and distribution of such nodes in a clustermay be selected by the customer or user so that each cluster providesthe desired number of computer, monitor, network, or other nodes as maybecome available. Advantageously, the nodes are implemented as plug-inor removable modules, such as printed circuit boards, so that theconfiguration of any particular cluster or of a system having aplurality of clusters may be modified after manufacture. In this wayadditional nodes of any desired type may be added when the need arises.Not all locations within a cluster need be populated thereby providinginitial cost savings as well as allowing later expansion. Nodes may bedynamic configured, either identical nodes or specialized nodes, aresupported in response to changing loading and QOS.

Recall that in the standard Integrated Server System (ISS) architectureincludes a single 2U (3.5-inch tall) box, has N nodes where in oneembodiment N=16. Internally there is a switching fabric that makesconnections between the nodes. The switching fabric may be a hub, aswitch, or any other means for making connections between all thedifferent the nodes. Internally, it is preferred to provide to suchswitching fabrics. This is advantageous (but not required) as it permitsimplementation and configuration to two separate and independentnetworks. For example, one network can connect multiple nodes of anytype and a second network can connect to data in mass storage units suchas may be used in a Storage Area Network (SAN). This is desirable insome circumstances as it reduces contention over the network and reducesthe likelihood of collisions of traffic over the network.

A second reason for providing two (or more) switching fabrics relates toproviding high availability or redundancy. High availability pertains toproviding the 24 hour/day 7 day/week (“24/7”) presence and availabilityover the internet. When only a single switching fabric and its set ofinterconnected nodes is used, a failure of that switching fabric or of acritical node not redundantly provided will fail to provide the high24/7 availability expected. Provision of two independent switchingfabrics and appropriately configured node sets provides either actualredundancy or the ability to either manually or automaticallyreconfigure either of the node/switch sets to maintain serviceavailability.

Therefore, it will be appreciated that the two (or other plurality)switching fabrics and their couple nodes may be used either as two (ormore) separate networks or maintained as a backup that assumes theresponsibilities of the primary set in the event of failure. Again, thisrollover from primary to backup may occur either manually orautomatically.

Typically, the two switching fabric means SW1 and SW2 in the embodimentof FIG. 4 will be identical, though they are not required to beidentical, and in at least one embodiment are implemented as separateprinted circuit boards that plug into the backplane of the cluster.

The inventive architecture also provides means for cascading orinterconnecting multiple clusters, and by implication, for cascading orinterconnecting the nodes in one cluster to the nodes in any number ofother clusters. Usually two such links are provided for coupling toother clusters, thereby allowing cascading of any number of clusters andnodes. For example, if each cluster box includes 16 nodes, connection toother clusters provides additional nodes. Cascading of any numberbetween two and twenty or more units may be provided. When multipleclusters are interconnected in this way required functionality mayoptionally be provided in only one cluster and need not be duplicated inall clusters. For example, if a monitor type node is desired it needonly be provided in one of the clusters to permit monitoring of all ofthe nodes of the connected clusters. Switching fabrics may alsooptionally be shared between interconnected or cascaded clusters.

In the embodiment of FIG. 4, the ISS standard architecture includes aComputer Node (CN) having a switching fabric that we call the NetworkNode (NN). The monitor node has a serial port that has a RJ-11 modembuilt in. In the event of a problem with the switch or any othercomponent, a page or phone call can be placed to a local or remoteadministrator with diagnostic information and allow the administrator tointeract with the cluster to take corrective action. For example, theadministrator may access local software diagnostic tools to troubleshoot and correct the problem, perform a hardware reset, perform a powercycle (OFF/ON) type reset, or otherwise debug, diagnose or correct theproblem.

Advantageously, but optionally, a separate monitor node (MN) is providedfor each switching fabric means even though either of the monitors maybe configured to monitor both switching fabrics any all of the nodescoupled to or through the switching fabric. This duplication is providedfor purposes of redundancy so that in the event that one of theindependent networks fails or the modem itself fails, the remainingoperational network may be monitored so that intervention by theadministration may be accomplished as desired. Also, in the event that amodem fails, modem redundancy allows the administrator to query eitheror both networks. It also facilitates a determination that a modem hasfailed versus the network having failed.

Physically, it is a rectangular rack-mountable box. In one embodiment,the 16-node ISS enclosure is provided as a standard 19-inch wide,3.5-inch high (2U) rack mountable chassis. Hot swapping any and all ofthe boards with which the nodes are implemented is supported. The boxneed never be powered down and therefore so long as a minimum set ofnodes remain in the box, the network remains available. There are 16computer node boards (also referred to as server modules) that may beplugged or unplugged at any time. Each board (computer node or servermodule) is coupled to the other nodes and to the switching fabric via abackplane bus so that no external cables or wires are required forconnecting the nodes within any cluster box. In preferred embodiments ofthe invention, the switch or switches are built into the box, though inother embodiments external switches, such as switches within a cascadedcluster, may be used. Where clusters are to be cascaded (see descriptionabove) the connections between cluster boxes may be made with externalcables. It will be appreciated that for a 16-node per cluster box thereduction in cables is substantial (up to 3 1 cables between nodes areeliminated).

It will therefore be clear to workers having ordinary skill in the artin light of the description provided here that the inventive structureand method provides numerous features and advantages over conventionalsystems and methods. For example, the invention provides a IntegratedServer System (ISS) comprising multiple nodes housed within a singleenclosure or box. In one embodiment, 16 nodes within a single enclosureare supported, but any number that may physically be placed within asingle enclosure may be used, including for example any number of nodesbetween 1 node and 32 nodes or more. Configurations having 4, 8, 10, 12,16, 20, 24, and 32 nodes are specifically provided. Larger numbers ofnodes may readily be accommodated if the size of the enclosure isincreased and due attention is provided for cooling or other heatdissipation. Nodes available in any particular enclosure may be selectedfrom network nodes (NN), computer nodes (CN), monitor nodes (MN), aswell as variations and combinations of these node types.

In another aspect, the inventive structure and method may betransformed, morphed, or otherwise configured to provide (either aloneor in combination with other cluster units) a great variety oforganizations and architectural topologies, and therefore provide analmost unlimited number of functional configurations.

In another aspect, all nodes within an enclosure are connected to eachother and to a switching means by a backplane bus internal to theenclosure, thereby eliminating the need for external node-to-node andnode-to-switch connection cables. Such conventional cables are prone tofailure and inadvertent disconnection during service operations that mayresult in network downtime. In yet another aspect, the inventivestructure and method facilitates and permits any node to perform anysupported function or operation. In one embodiment, all nodes areidentical and can be adapted, such as by programming or loadingappropriate software, to provide any function or operation. In anotherembodiment, different classes or types of nodes are provided that aresomewhat specialized and/or optimized to perform selected classes offunctions or operations very well. In yet another embodiment, highlyspecialized nodes are available to perform specific functions. In eachof these embodiments, the nodes are desirably provided as removablehot-pluggable modular units, such as PC boards or cards, that may beadded or removed from the enclosure without powering off or otherwisemaking the network unavailable. This facilitates the interchange of hotspares which may remain ready and available within the enclosure forimmediate use in the event of a node failure. In still another aspect,each Integrated Server System (or cluster) unit is cascadable so thatmultiple sets of nodes may be interconnected to provide the desirednumber and type of operation. In yet another aspect, any and all nodesare reconfigurable at any time based on such factors as load or qualityof service (QOS) requirements. Furthermore, the change orreconfiguration may be communicated to other nodes and the effect ofsuch reconfiguration ripple through to the other nodes and to thenetwork as a whole. This permits the entire system to be self balancingto the extent desired. In another aspect, each cluster is provided withsufficient intelligence so that at least some network administrationoperations that conventionally required some degree of supervision orintervention may be performed autonomously and dynamically in responseto sensed conditions experienced on the network or within one or morenodes of the network.

In still another aspect the inventive structure and method provide forsignificant power consumption reduction and energy savings as comparedto conventional network and server architectures as only those powerconsuming resources that are actually needed to provide the quality ofservice required are in an active mode. Those node resources that arenot needed may be powered off or placed in some power conserving standbymode until needed. In addition, operations performed by one or morenodes may be shifted to another node so that only the remaining activenodes consume power and the remaining nodes are in standby mode orpowered off until needed. The intelligence within one of the nodesacting as a master node for the cluster or ISS may then wake up theinactive node and configure it for operation. A system may be woken upand placed in any of the available operating modes by any one of aplurality of events. Nodes may also be placed into an inactive or powerconserving mode when no demands are made on their resources independentof whether responsibility for their functionality has been shifted toanother node or nodes. In one embodiment of the invention the powerconsumed is reduced by a factor of about 10-times as compared to astandard 19-inch wide by 1.75-inch high (1U) rack mountable network nodedevice. This power savings is accomplished at least in part by one ormore of the following measures: the reduction in the number of powersupplied, use of the mounting plate as a heat sink to assist in removingheat from the enclosure, providing power saving controls to circuits anddevices within the ISS enclosure, and the above described ability toreconfigure and take off line unneeded capacity.

The architecture is referred to as the Integrated Server System (ISS) orthe integrated server architecture, and each unit is referred to as anIntegrated Server System Unit. One embodiment of the ISS Unit is beingdeveloped by Amphus under the proprietary name Virgo™.

Having now described a first embodiment of the Integrated Server System(ISS) (also referred to as the Integrated Server Architecture),attention is now directed to several further embodiments which aredescribed in somewhat greater detail so that the advanced powerconsumption reduction features may be more readily understood.

An exemplary embodiment of an ISS based system is illustrated in FIG. 6.Each Integrated Server System (ISS) architecture comprises a number offunctional components. A particular exemplary embodiment is nowdescribed although it will be clear from the description provided thatvarious changes to the configuration may be accomplished withoutdeparting from the spirit and scope of the invention. In this embodimenta chassis and/or enclosure houses a backplane mounting a plurality ofconnectors adapted to receive a plurality of printed circuit boards. Thenature, type, characteristics, and number of these printed circuitboards 107 may vary from installation to installation as will bedescribed subsequently. It will also be appreciated, that the physicalform and/or connectivity of these components may be through other means.

In one embodiment of the invention, multiple ISS units may be coupledtogether or interconnected. In the embodiment illustrated in FIG. 6 twosuch ISS units 102 are shown. A first of these is referred to as the“A-unit” and the second unit is referred to as the “B-unit”. Additionalunits, may also be provided. It is noted that although theconfigurations of the A-unit and B-unit are the same here, in anypractical implementation, they may be the same or different, dependingupon a functional purpose all of the overall system, and/or all ofindividual modules within the system. The manner in which configurationsare chosen, physically altered such as through the addition or removalmodules, and/or through dynamic allocation of modules are made inaccordance with principals described hereinafter. With this in mind,components resident within the a-unit are typically designated with an“a” suffix to the reference numeral and be components resident withinthe bee-unit are typically designated with an “b” suffix to thereference numeral. However, where a general reference to a component ofa particular type is made without specific reference to diagram, the “a”and the “b” suffix may be dropped for convenience.

Each ISS units also comprises at least one, and generally a plurality,of server modules 112 a-1, . . . , 112 a-N, where in a particularembodiment of the ISS the maximum number of server modules 112 is fixedat 16 due to current physical size constraints of the chassis. Each ISSmay also included one or a plurality of management modules 108 a-1, . .. , 108 a-M, where in a particular embodiment of the ISS maximum numberof management modules is two. It should be understood about thatalthough each ISS unit may include one or more management modules 108,management functionality may alternatively be delegated to managementmodules physically residing within other ISS units so that themanagement module functionality of any particular ISS unit may resideelsewhere.

In one implementation, the integrated server system includes at leastone primary switching fabric 104 a-1 also referred to as a primaryswitch module, and advantageously includes a secondary switching fabricor secondary switch module 104 a-2. The first (sometimes referred to asthe primary) switch module 104 a-1 operates to connect for communicationeach (any and all) the modules that are present in the ISS Unit, such aseach of the Server Modules, Management Modules, Power supplies, coolingunits, and any other module or unit that might be present. Having thesecond (or secondary) switch module 104 a-2 operates to provide the samefunction as the first module as well as providing a redundantcommunication path between and among the modules or other units that arepresent in the ISS. Therefore while a second (or secondary) switchmodule is not required for any particular ISS, the presence providessignificant benefits in high-end applications.

Each switch module provides a multi-connection switching fabric to linkthe modules with one another. In one embodiment, each switch has theequivalent of a switching matrix inside that establishes connectionsbetween different modules. For example, one or more of server modules,management modules, power supplies, fan modules, may be coupled togetherfor communication. More particularly, the switch module may connectmanagement module 1 with any of the server modules (for example withserver module 5) or with the other management module, power supplymodule, fan modules, or the like. In general, the switch module makesone or a plurality of direct connection and is not typically implementedas a bus architecture that would allow only dedicated use by a singledevice or module (or a pair of communicating devices or modules) at anyparticular time. Switch module permits multiple simultaneouscommunication without collision.

One or a plurality of server modules (SM) 112 are also provided. Servermodules are operative to serve data or other content in a manner that iswell known in the art and not described in greater detail here. Forexample, a server module may be configured so as to enhance, improve, oroptimize serving web pages, cached data or content, streaming video, orother data or content types as is known in the art. Server module harddisk drive configuration parameters that may be adjusted or modifiedaccording to the type and quantity of data or other content to beserved. Such configuration and configuration utilities are known in theart, and include but are not limited to the data organization on theserver hard disk drive (such as a modified RAID data organization andthe RAID level).

Each SM 112 is advantageously-implemented as a printed circuit (PC)board or card having an edge connector (or electrical contacts) adaptedfor plug-in connection to a mating receiving connector associated with achassis backplane board. An SM also includes a PC card mountedprocessor, such as a microprocessor, microcontroller, or CPU, andassociated memory. At least one mass storage device, such as a rotatablemagnetic hard disc drive, optical drive, solid state storage device, orthe like is mounted to the PC card and coupled to the processor. Themass storage device provides storage of the data or content to beserved, or information concerning a location or link at which the dataor content may be found if it is not served directly from the particularSM 112. While physical, functional, and operational aspects of theserver modules are novel, especially in the areas of power consumptionand power management, data or content throughput control (QoSthrottling), heat dissipation and cooling, mass storage devicecharacteristics, form factor and the like, the manner in which data orcontent is stored and served is generally conventional in nature, andnot described in greater detail here.

A management module (MM) 108 is operable to provide overall ISSUmonitoring and control. These management and control functions aredescribed in greater detail in the context of the power managementfunction. In general, each ISS unit will contain at least one MM 108 andin high-performance implementations and where redundancy is desired,each ISSU will include multiple MMs. In one embodiment of the ISS, twoMM are provided. In such implementations, the two MMs may shareresponsibilities or more typically the second MM 108 a-2 will provideredundant backup for the first MM 108 a-1. Management Modules 108 aredescribed in greater detail in a elsewhere in this description.

At least one, and advantageously a plurality of temperature sensors aredisposed within the ISS enclosure. Each of these temperature sensors aredesirably located at diverse locations within the enclosure so that thetemperature of heat sensitive components may be adequately monitored andcorrective action taken as needed. These diverse locations may beselected from locations on the internal surface of the enclosure,locations on the chassis, locations on one, more than one, or all of theserver modules, management modules, switch modules, power supplymodules, fan modules, or back plane, and may be integrated within solidstate devices such as within the CPU.

In one embodiment of the invention, a fully populated ISS Unit havingsixteen server modules, two management modules, two switching modules,two power supplies, two fan modules, and the backplane that supportsthese components, includes about 30 temperature sensors. Here eachserver module includes one temperature sensor integrated in the CPU andone on the edge connect board that supports the CPU and other circuitryas well as the hard disk drive. There is also at least one temperaturesensor on each management module. While some embodiments may providetemperature sensing of the chassis, enclosure, or backplane, in thepreferred embodiment no such temperature sensors are provided in theselocations for reasons of reliability. As described in detail elsewherein this specification, the preferred embodiment of the ISS Unitbackplane does not include any active components. It merely providesprinted circuit traces that provide electrical operating power (voltagesand current) and communication, as well as providing physical supportand connectors that receive the edge connector (or other) plug inmodules.

In one embodiment, the temperature sensors have a preset temperature atwhich an output signal changes state so that they effectively generatean over temperature signal, in another embodiment the temperaturesensors 150 generate a signal that indicates a temperature ortemperature range. Sensors on different devices and/or at differentlocations may be of different types and/or the circuitry (for hardwarebased sensing and control) and/or algorithm (for sensing and controlinvolving software or a computation element as well as hardware) mayprovide for different response to a particular temperature. Temperatureawareness and control for an ISS Unit (ISSU) may even involve controlbased on multiple sensors, temperature differences, and/or a time rateof change of temperature.

Different physical device types may be used as well. For example,temperature sensors 150 may include a temperature sensor (such as forexample a thermistor, thermal-couple, or other devices known in the artthat have an electrical characteristic that changes with temperature.)Mechanical or electro-mechanical sensors such as sensors that usebimetallic switches to oven and close a connection may be used. In oneembodiment, temperature sensing circuitry is integrated into a PC boardmounted component or as a surface mounted component on the PC board ofthe server modules, management modules, switch modules, or othercomponents of the ISS.

Independent of the form or the temperature sensor, the signals generatedby the sensor or circuitry associated with the temperature sensorsprovide signals (analog or digital) to a management module (or a servermodule adapted to provide some management function) so that theintelligence built into the management module may control theoperational parameters for one or more head generating elements (forexample, the server, management, or switch modules) and the heatdissipating elements (for example, the fan modules or the individualfans within the or each fan module.)

Each ISS also advantageously includes dual redundant fan modules 114 a,each of the modules including a plurality (typically two) of fans orother heat absorption or heat dissipation devices. Such cooling may beaccomplished by conduction, convention, or radiation generally. Air orother fluid flow may be used. In one embodiment each fan module includesfirst 114 a-1 and second 114 a-2 electric motor driven fans.

Dual redundant fan modules 114, each having one or a plurality of fans,are advantageously provided so as to accomplish the required coolingfunction, at a reduced or minimized power consumption level, to providecooling system redundancy, and to support hot-plug maintenance and/orreplacement of the fans and fan modules. The manner in which ISS powerconsumption is reduced using this fan and fan module configuration aredescribed elsewhere in this description.

Each ISS 102 includes at least one power supply, advantageouslyimplemented as a hot-pluggable replaceable power supply module.Desirably, an ISS includes two such or dual-redundant power supplymodules so as to provide sufficient power or energy for operating theswitch module(s) 104, management modules 108, server module(s), and fanmodules 114 within the ISS 102 as well as connected components that maydraw power from the ISS. Power consumption and control aspects of thepower supplies are described in greater detail elsewhere in thisdescription.

A backplane providing operating power (for example, one or more of ±3Volt, ±5 Volt, ±12 Volt depending upon the voltage and currentrequirements of the modules, and ground), communication (such as in-bandand out-of-band communication via Ethernet, serial interface, and/orother interface) is mounted in chassis. The backplane also providescircuit protection in the form of circuit breakers or other over currentor over voltage protection devices to protect the backplane traces andthe modules that are or may be connected at the time of an undesiredelectrical component failure or other hazardous or damaging event.Protection may also be provided either in conjunction with the backplaneor the modules themselves for under current or under voltage conditions.

A plurality of appropriately sized and shaped electrical connectors (forreceiving PC board based edge connectors are disposed on the backplanePC board to connect to the management modules, server modules, andswitch modules. The fan modules, power supply modules may coupledirectly to the backplane or communicate with backplane coupled modules(such as the management module) via separate couplings. In conventionalmanner, the chassis includes guides or slots that assist in physicallylocating and guiding the different modules or other componentsphysically in place to make secure electrical contact with the matingconnectors.

In a preferred embodiment of the invention, each ISSU includes abackplane in the form of a multi-layer printed circuit board that isdevoid of active electrical circuit components. This increases thereliability of each ISSU and the system as a whole. It is noted that apreferred configuration of an ISSU provides multiple redundanthot-swappable server modules, management modules, power supplies, switchmodules, and fan (cooling) modules. In such a configuration, there is nosingle point of failure as redundancy is provided everywhere. As onlyone backplane can reasonably be provided within an ISSU, only electricaltraces (or wires) are provided on the backplane. In a preferredembodiment, no electrical circuit components are present and onlyelectrical traces (and connectors) are present. While an ISSU havingconventional backplane technology may be used to achieve the powersaving benefits described throughout this specification, the inherentredundancy and reliability of the ISSU would be compromised byconventional backplane technology that incorporates active failure-pronecircuit elements. For example, if a backplane failed in suchconventional implementation, the unit would need to be powered down andall modules removed so that the backplane could be replaced. There areno other daughter boards other than the ones described. There are onlyconnectors and traces, because active components could not be replacedwithout downtime.

All components are hot swappable to the backplane. For a sixteen servermodule configuration, it is desirable that a failure of any one notnegatively impact the operation or performance of any other. (Of coursecontrol is provided for surviving server modules, management modules,switch modules, fan modules, and power supply modules to recognize afailure of another module or component and provide backup operationuntil the failure is corrected. Even with respect to power delivery,there is a separate set of traces and circuit breaker, fuse, or othercircuit protection for every plug-in module (server, management, switch,and fan or cooling). For example, without such separate power plane foreach module, if one server or other module were to short-circuit itwould take down all of the other modules in the ISS Unit or box. It isnoted, that even the failure of a capacitor within a circuit of a servermodule may act as a short circuit and that such capacitor failures maycommonly occur. Each power plane for the servers are separate andisolated from one another. The inventive backplane and moduleconnectivity protects the integrity and operation of the system fromeven direct short circuits. Also, since there are no active componentsin the backplane, the failed module is merely replaced and operationcontinues without need to repair or replace the backplane.

A serial interface 142 is preferably but optionally provided to supportan alternative communication channel to the back plane bus between andamong each of the server modules 112, management modules 108, switchmodules, or other modules or units, as well as to certain externalelements or components such as to a local management node 138 whenpresent.

The provision of the serial communication channel is advantageous as itprovides out-of-band communication should the in-band link (for examplethe ethernet link) fail. It also permits multiple alternative redundantcommunication. Diagnostics, console operations, and other conventionalcommunication may also be provided. Communication via the localmanagement mode or via a dial-in session are supported. The switchmodule(s) 104 may also be coupled to the management modules and theserver modules as well as the external elements or components via thesame serial bus or connection.

In one embodiment the serial bus provides an alternate communicationchannel. While this alternate communication channel is provided as aserial communication channel provided in one embodiment, it isunderstood that this represents a low cost and efficient implementation.Those workers having ordinary skill in the art will appreciate thatvarious types of alternate communications channels or links mayalternatively be provided, such as for example a Universal Serial Bus(USB), and IEEE 1394 (Fire Wire), or the like as are known in the art.

In a preferred embodiment, the serial interface architecture providestwo serial ports for each of the sixteen server modules. Each managementmodule picks off all two pairs from the sixteen and multiplexes theminto a single physical outlet or connector, this is referred to as theAMPC architecture that includes the AMPC bus.

In one embodiment, now described relative to FIG. 7, the AMPC Busprovides a communications channel for communicating serial data, andvideo, as well as keyboard and mouse inputs. Typically, the serial dataand any video data flows from one of the plurality of Server Modules tothe Management Module(s) and the keyboard and mouse input or commandsflow from the Management Module(s) to the Server Modules. Ethernet andserial I/O (SIO) connections are also provided to and from theManagement Module for redundancy and alternative access.

This time-domain or time-sliced multiplexing and selection eliminatesthe need for so many physical connectors. Each Management Module has aselector for one of the 32 (2×16) serial lines, and places the selectedserial pair on the single Management Module connector. Of course,multiple connectors either with or without some level of multiplexingmay be provided, but such configuration is not preferred as it wouldlikely increase the physical size of a Management Module unit anddecrease the effective density of the ISSU. Besides the serialinterface, keyboard, video, and mouse (KVM) data or signals can betransferred to and/or from the Management Module using the same or asimilar scheme.

A load balancer 128 couples each ISS unit 102 via an uplink, such as avia a gigabit uplink, to a router 130. The load balancer 128 is ofconventional and includes intelligence to sense the load on each of theoperating servers and task the servers according to some predeterminedrules or policy to serve data or content. When used in connection withthe inventive power conserving features, the intelligent load balancerand router are operative to sense which of the server modules are in anactive mode and to route server tasking to those active server modulesaccording to some policy. Policies concerning how many server modulesshould be maintained in an active mode, what CPU core voltage and clockfrequency such active server modules operate at, and other server moduleoperating characteristics are described elsewhere herein. Router 130 isinterposed between the load balancer 128 and a network of interconnectedcomputers or information appliances, such as for example the Internet132. Though advantageously provided, where appropriate, load balancersand/or routers may be eliminated. For example, they would not berequired when only a single server module is provided. The structure andoperation of load balancers 128 and routers 130 as well as the Internet132 are well known and not described in further detail here.

The bi-directional uplinks (and downlinks) 122, 124, 126 arecommunication links that provide high-capacity, high-throughput datacommunication between the ISS 102 (actually the switch module 104 of theISS) and the external world, including the load balancer 128 and theNetwork Attached Storage (NAS) 120. Gigabit uplinks for uploading (anddownloading) data or content provide high data rate communications andare known in the art and therefore not described in greater detail here.Alternatively, an up and down link can be aggregated to provide twouplinks as illustrated in FIG. 8, which shows a plurality of ISSU (ISS₁,ISS₂, . . . , ISS_(n)) coupled to first and second Gigabyte switchesGS1, GS2. Gigabyte switch GS₁ is coupled to a router which is in turncoupled to a network, such as the Internet. Gigabyte switch GS₂ may besimilarly coupled.

Network Attached Storage NAS is optionally but desirably provided forseveral reasons. While the storage provided for each server moduleprovides rapid access and response to requests, the size of the servermodule may necessarily limit the amount of data available on anyparticular server module. For example, 2.5-inch and 3.5-inch form factorhard disk drives may typically have capacities in the range of32-Gigabyte to 100-Gigabyte of storage, though such capacity may beexpected to increase as new recording media and head technology aredeveloped. In any event, NAS in the form of one or more hard diskdrives, RAID arrays, disk farms, or the like mass storage devices,arrays, or systems provide substantially greater storage.

Content that has been requested or that will be requested and servedwith high probability may be uploaded from NAS to one or more servermodules and cached for later serving. Another benefit of the attachedNAS is that a single copy of data is provided that is accessible to allthe server modules and can be accessed either directly when only one ispresent, or through a switch when more than one is present. It is notedthat the switch module coupling the ISSU to the load balancer isdifferent than the switch module from the ISSU to the NAS.

Alternative access nodes and connectivity are provided for monitoringand managing operation and configuration of a particular ISS, componentor module of an ISS, or ISS and/or components coupled to an ISS forwhich monitoring or management are desired. In one embodiment, thisaccess is provided by a remote internet management node 136 coupled viaan internet connection 134 to the internet 132 and hence via router 130,optional load balancer 128, and uplink/downlink 124, 126 to the ISS 102.Within each ISS 102, monitoring and/or management operations willtypically be carried out by a defined communication path (typically overthe backplane) to one or more of the management modules 108. It is notedthat the backplane provides multiple sets of traces for multiplecommunication channels, including ethernet and serial channels, and thatthe backplane is not limited to a single bus. Monitoring and managementaccess from remote Internet management node 136 over an Internetconnection 134 is desirable as it provides additional redundancy andconvenient monitoring and control using readily available protocols fromvirtually any remote location.

An alternate path is desirably provided to a local management node 138over the serial communications channel 142, and a second alternate pathmay desirably be provided from the local management node 138 to one ormore of (and preferably to all of) the management modules over a secondethernet communication channel or link 140 that is different from theethernet control channel. Monitoring and management access from localmanagement node 138 over ethernet communication link 140 is desirable asit provides another alternative connection, communication, and possiblecontrol when desired, and advantageously permits connection usingstandard TCP/IP software and protocols. A further alternatecommunication path may desirably be provided via a remote dial-inmanagement node 146 over a Plain Old Telephone Service (POTS), typicallytrough the local management node 138, and then either over the ethernet140 or the serial connection 142. While communication with the ISS overany of these communication channels may itself suffice, the provision ofalternate links and communication schemes provides for considerableflexibility in access, management, and control. The alternate paths alsoprovide considerable redundancy from single channel failure in order todiagnose and service the ISS or ISS-based system in the event of afailure. For example, should a problem occur that disables the switchmodules 104 and access via the gigabit uplink/downlink paths 124, 126,communication with the management modules 108 and with the rest of theISS will still be possible on site either over serial bus 142 orethernet link 140. When access from a remote location is desired, eitherdial-up (such as via a phone modem) or Internet based access isgenerally; however, each serves as a redundant alternate path for theother in the event of failure.

It is particularly noted that the integrated structure of these ISSunits provides a small form factor (2U high chassis/enclosure); highserver module density (sixteen server modules per ISS in oneembodiment); switch module, cooling/fall module, power supply module,management module, and server module hot plug-and-play and highavailability via redundancy; lower energy or power consumption thanconventional servers; and many other advantageous features as describedin greater detail herein.

Many different types of servers architectures are known in the art.Typically, such servers have at least one processor with associated fastrandom access memory (RAM), a mass storage device that stores the dataor content to be served by the server, a power supply that receiveselectrical power (current and voltage) from either a battery or linevoltage from an electrical utility, a network communication card orcircuit for communicating the data to the outside world, and variousother circuits that support the operation of the CPU, such as a memory(typically non-volatile ROM) storing a Basic Input-Output System (BIOS),a Real-Time Clock (RTC) circuit, voltage regulators to generate andmaintain the required voltages in conjunction with the power supply, andcore logic as well as optional micro-controller(s) that communicate withthe CPU and with the external world to participate in the control andoperation of the server. This core logic is sometimes referred to as theNorthbridge and Southbridge circuits or chipsets.

From a somewhat different perspective, variations in serverarchitecture, reflect the variations in personal computers, mainframes,and computing systems generally. The vast structural, architectural,methodological, and procedural variations inherent in computer systemshaving chips, chipsets, and motherboards adapted for use by IntelProcessors (such as the Intel x86, Intel Pentium™, Intel Pentium™ II,Intel Pentium™ III, Intel Pentium™ IV), Transmeta Crusoe™ with LongRun™,AMD, Motorola, and others, precludes a detailed description of themanner in which the inventive structure and method will be applied ineach situation. Therefore in the sections that follow, aspects of theinventive power management and ISS system architecture are describedfirst in a general case to the extent possible, and second relative to aparticular processor/system configuration (the Transmeta CrusoeProcessor). Those having ordinary skill will appreciate in light of thedescription that the inventive structure and method apply to a broad setof different processor and computer/server architecture types and thatminor variations within the ordinary skill of a practitioner in thefield may be made to adapt the invention to other processor/systemenvironments.

Before describing particular implementations that relate to more or lessspecific CPU designs and interfaces, attention first directed to asimplified embodiment of the inventive system and method with respect toFIG. 9. In this embodiment, at least two (and up to n) server modules402-1, . . . , 402-N are provided, each including a CPU 404 and a memory408. CPU 404 includes an activity indicator generator 406 whichgenerates activity indicators, and either (i) communicates the activityindicators to memory 408 for storage in an activity indicator(s) datastructure 410, or not shown, (ii) communicates them directly to a servermodule control unit and algorithm 432 within management module 430.Different types of activity indicators such as are described elsewherein the specification, such as for example an idle thread based activityindicator may be used. Whether stored in memory or communicateddirectly, the activity indicator(s) are used by the management module todetermine the loading on each of the server modules individually and asa group. In one embodiment, activity information or indicators createdon any one computer or device (such as a server module) is accessible toa manager or supervisor via standard networking protocol.

Although not illustrated in FIG. 9, analogous structure and signalsgenerated and received may be used to control the operation of corelogic circuits to thereby control core logic voltage and core logicclock signals in a manner to reduce power consumption where such corelogic power management is provided.

Voltage and frequency are regulated locally by the CPU using an activitymonitoring scheme, such as for example one of the activity monitoringscheme illustrated in Table I. TABLE I Exemplary Activity MonitoringSchemes carried out in CPU or PMU Carried Carried out by CPU out by PMUApplication Layer Port Address NA Network Layer TCP/IP NA Physical LayerIdle Threads, I/O Activities Activity Counter

This power management scheme may be interpreted in one aspect asproviding a Mode1-to-Mode2 and Mode2-to-Mode1 power management scheme,where both Mode 1 and Mode2 are active modes and the state of the CPU ineither Mode 1 or Mode 2 is controlled locally by the CPU, and in anotheraspect as providing a Mode3 (inactive mode or maintenance of memorycontents only). Mode3 control may also be performed locally by the CPU,but in one of the preferred embodiments of the invention, entry into aMode 3 stage is desirably controlled globally in a multi-CPU system.Where the multi-CPU's are operative with a plurality of servers formulti-server power management, the Management Module (or a Server Moduleacting as a manager on behalf of a plurality of server modules)determines which Server Module should enter a Mode 3 state using theServer Module control algorithm and unit 432. Activity monitoring ofindividual Server Modules 402 is desirably based on the standard networkprotocol, such as for example SNMP. Therefore the activity indicatorsmay be retrieved from the CPU 406 or memory 408 via NIC 440 as is knownin the art. A communication link coupling microcontrollers (μC) 442together, and in particular the microcontroller of the Management Modulewith the microcontrollers of the several Server Modules. This permitsthe management module to communicate commands or signals to the servermodules which are received by the microcontrollers even when the CPUsare in a suspended state (Mode 3). In so providing for monitoring overthe first link (the Ethernet) and control over the second link (the AMPCbus), the server modules may be monitored for activity and controlledglobally to reduce power consumption while providing sufficient on-linecapacity. It is noted that the power management may be effected byaltering either or both of the CPU clock frequency 420 or the CPUvoltage 416.

Although a separate management module 430 is illustrated in FIG. 9, itshould be understood that the management functionality generally, andthe server module control algorithm in particular may be implemented byone of the operating server modules. For example, the control algorithmwould be implemented as a software or firmware procedure executing inthe CPU and processor of a server module designated according topredetermined rules, policies, or procedures to be the master.

It is noted that although several of the modes described conserve power,they do not compromise performance, as the cumulative combination ofserver modules is always maintained at or above minimum targetedperformance.

In FIG. 10 there is illustrated an exemplary system 301 including aserver (such as for example, an ISSU server module) 302-1, coupled to aswitch (such as for example, an ISSU switch module) 304, and through theswitch 304 and optionally via a micro-controller (μC) 314 within server302 over a separate (optional) direct bus connection 312 (such as forexample, the AMPC bus made by Amphus of San Jose, Calif.) to a powermanagement supervisor (such as for example, ISSU management module) 316.As described elsewhere herein, switch 304 is responsible for connectingthe various server module(s) 302, management module(s) 316, and othercomponents that are or may be controlled to achieve the powerconservation features of the invention. Recall that such subsystems asthe power supply (not shown) and cooling or fan modules may also becoupled through the switch 304. The connectivity and signals shown inthe diagram are intended to show significant control paths pertinent tothe operation of the invention, and therefore some signals that areconventional or do not illustrate the operation of the invention are notshown to avoid obscuration of the invention.

Attention is now focused on the internal structure and operation of theserver module 302. During operation CPU 320 executes commands orinstructions, or when no instructions are present to be executed,executes idle threads. The activity level of the CPU is monitored and acontrol signal Vcc_CPU_control 322 is generated based on that sensedactivity or lack of activity. The manner in which this activity issensed or the manner and characteristics of the Vcc_CPU_control signalwill typically vary depending on the processor type, operating system,and other factors specific to the system architecture. By way ofillustrative example, an indication as to the CPU activity or lack ofactivity may be generated by monitoring by executing an applicationlayer function call that returns a value indicating the idle threadexecution based activity. This is possible in the Microsoft Windows 98,2000, and NT operating environments, for example.

As the name implies, the Vcc_CPU_control signal 322 which is an inputsignal to voltage regulator 324 controls or influences the CPU corevoltage Vcc_CPU 326. As described elsewhere in this description, the CPUcore voltage 326 may be raised and lowered in conjunction with the CPUclock frequency to provide adequate switching response of CPU circuitswithout excessive voltage. Although this embodiment illustrates that theVCC_CPU_control signal 322 is generated within the CPU, in analternative embodiment, it may be generated within the core logic block330. In one embodiment, the CPU clock is adjusted based on a signal fromthe core logic and the CPU voltage is adjusted on the basis of the CPUitself. This is due to the fact that the voltage change is desirablysynchronized in time with the frequency change. In some sense, thiscontrol may be viewed as including an effective link from the core logicto control the voltage regulator output.

Core logic 330 includes a Power Management Unit 332 of which many typesare now known; however, one early example of a Power Management Unit isdescribed in co-pending U.S. patent application Ser. No. 09/558,473 aswell as in U.S. Pat. Nos. 5,396,635, 5,892,959 and 6,079,025 (each ofwhich is herein incorporated by reference) by the inventor of thepresent application as well as in the other applications relatedthereto. In operation, PMU 332 receives a signal over bus 336 andgenerates an output signal 338 that is communicated over bus 340 toclock generator 342. Clock generator block 342 includes circuitry thatgenerates a CPU clock 50, a core logic clock signal 352, a NetworkInterconnect Card (NIC) clock signal 354, and a video clock signal 356.

RAM 328 is coupled to core logic 330 via DRAM control line and hence tothe CPU via bus 336. Hard disk drive 338 is similarly coupled to corelogic 330 to CPU via bus 336. In one implementation, Redundant Array ofIndependent Disc (RAID) data storage is provided for the server modules.As is known, this RAID storage provides considerable data redundancy. Inorder to implement this RAID in a power management efficient manner, twoIDE controllers (or enhanced IDE controllers) are used to interface totwo separate disk drives. Provision of two hard disk drives supportsRAID Level 0, RAID Level 1, and RAID Level 0+1 implementations. Aspectof the RAID power management disk drive longevity are described inco-pending U.S. Provisional Application Ser. No. 60/236,062 entitledSystem, Apparatus, and Method for Power Conserving and Disc-Drive LifeProlonging RAID configuration filed 27 Sep. 2000, hereby incorporated byreference. It is noted that providing RAID storage or multiple diskdrives on the servers is advantages though not required.

Clock generator 342 includes clock signal generating and logic circuitryor other means for generating a CPU clock signal at the desiredfrequency or for selecting a CPU clock signal from an availableplurality of clock signal having different frequencies. Under theinventive power management scheme, the clock frequency is adjusteddownward within a permissible CPU clock frequency range to provide a CPUprocessing power that matches the present need, and to the extent thatthe present need is below the maximum capability of the processor whenoperating at full permissible clock frequency, to reduce the powerconsumption of the CPU. As the CPU core voltage may be reduced below amaximum voltage when the clock frequency is below its maximum frequency,the CPU core voltage may be lowered with the clock frequency or speed.

A PCI bus 360 coupling NIC 362 and Video processor 364 is provided andinterfaces with CPU 320 via Core logic 330. NIC 362 generates andprovides a resume output 366 and NIC Clock input signal 368, and Videoprocessor 364 is provided with a video clock signal 356 from the clockgenerator 342 and a suspend input signal 370. It is noted that thesuspend and resume signals may come from multiple sources to affect thedesired control and management.

In this illustrative embodiment, an X-bus 374 is provided to couple theReal-Time Clock (RTC) 376 and BIOS 378 to the core logic 330 and via bus336 to the CPU as required. RTC 376 may generate a resume output signal39. This RTC generated resume signal 379 is therefore operative toactivate PMU 332, core logic 330, and CPU 330 under a predetermined timeor alarm condition. For example, the RTC may be set to generate a resumesignal 379 at 8:00 am local time every day to bring the server module302 back online.

The NIC resume signal may be generated when a specific packet isreceived. When generated in one of these manners and communicated to thePMU 332 it is operative to place the core logic 336 back into an activestate and hence CPU 320 into any selected state of mode. One situationin which the NIC resume signal may be generated is when the servermodule is in a powered-on but inactive state, such that the CPU clock isstopped (or operating at an extremely low clock frequency). Under suchcondition, a simple way of waking the server module 302 is tocommunicate a signal 380 from management module 316 via switch 304. Asthe NIC will typically be kept active, it will receive the signal 380and generate the resume signal 366.

It is noted that each of the elements, such as the hard disk drive,Video processor and other power consuming elements may include means forreceiving a control signal that places them into a power conservingstate or that brings then out of on or more power conserving states intoa full power and performance mode.

It is noted that the embodiment illustrated in FIG. 10 represents asystem that might utilize any of a number of conventional processors orCPU, and might for example utilize a CPU of the Intel Pentium, PentiumII, Pentium III, or Pentium IV types made by Intel Corporation of SantaClara, Calif., various Advanced Micro Device CPUs, CPUs made byTransmeta, as well as other processors and CPUs as are known in the art.

Having now described two generic systems and methods for power managinga server having at least one CPU, attention is now directed to FIG. 11,where is illustrated a functional block diagram of an embodiment of anserver module 102 adapted for use with the Transmeta Crusoe processorhaving the LongRun internal power management feature. The Transmeta CPUchip design is identified separately only because it provides for thechoice of several (actually 15) different Mode 2 operating levels havingdifferent CPU clock frequency and CPU core voltage combinations.

For one of the inventive server modules, each server module includes atleast one processor such as a CPU 201. Other embodiments provide formultiple CPUs however for the sake of simplicity, the descriptionfocuses on single CPU configurations. CPU 201 includes a plurality offrequency control registers 205. The frequency control registers areloaded with values used to control the clock frequency at which to CPUcore runs. A configuration ROM 202 coupled to the CPU is operative toprovide a basic input output system during a CPU boot process. A CPUtemperature sensor 204 is also coupled to CPU 201 and is operative tomodify the values stored in the frequency control registers in responseto a sense to CPU temperature so that CPU temperature is maintainedwithin acceptable operating limits. The CPU temperature sensor 204 alsocommunicates with power management unit (PMU) 224 which is itself partof the South Bridge unit 223. PMU 224 also receives a 32 KHz signalwhich is used for the real-time clock within the PMU. The PMU 224 isidentified as a component or sub-system of the South Bridge Unit 223,though in fact each may be implemented as a separate unit. Structure infunction of PMU 224 and South Bridge 223 are described in greater detailhereinafter.

CPU 201 is coupled to a memory 208, such as a 16-bit synchronous dynamicrandom access memory (×16 DDR SDRAM), via bus 210. Memory 208 alsoreceives a clock signal (DDR-CLK) 209 generated as an output callsfrequency control registers 205. This clock signal 209 is generated fromvalues in the frequency control registers. In one embodiment beprimarily SDRAM is soldered on board. Additional or expansion RAM 221may optionally be provided and is coupled to the CPU via bus 214.Optional expansion RAM 221 receives a clock signal (SDR-CLK) 212 alsogenerated by circuitry from frequency control registers 205. The DDR RAMis a higher performance memory than the SDR RAM and must be adjacent tothe CPU as the result of bus path length issues associated with the DDRmemory. It is for this reason that DDR memory cannot be used in anexpansion slot and the main and expansion memory are treated somewhatdifferently.

CPU 201 also receives a CPU clock signal (CPUCLK) 213 from a clockgenerating unit 232. Clock generator unit 232 receives a clock controlsignal 229 from the PMU 224 component of the South Bridge Unit 223. Thisclock control signal 229 is generated within the PMU. Clock generatorunit 232 also generates a plurality of additional clock signals. Theseinclude a first PCI clock signal (PCI-CLK1) 216 operating at a firstfrequency or rate which is output and communicated to an input port orpin of CPU 201 to control the clock rate or switching rate, a second PCIclock signal (PCI-CLK2) 230 operating at a second frequency or ratewhich is output or communicated to Video Subsystem 236, and a third PCIclock signal (PCI-CLK3) 231 which is output by the clock generator unit232 and communicated to the Ethernet Controllers Unit 233. Ethernetcontrollers 233 are desirably fast ethernet controllers capable ofoperating at a high data rate. Ethernet controllers 233 generate anoutput signal 235 that is communicated to a Transformer unit 234.Transformer unit 234 receives this input and generates an output in theform of a signal 257 and places this signal 257 on an ethernet bus 265via backplane connector 103. Ethernet controller Unit or NetworkInterface Card (NIC) 233, may contain one or a plurality of ethernetcontrollers, receives a Wake-On-LAN (WOL) signal 227. This WOL signalcauses the NIC 362 to generate a resume signal which is sent orotherwise communicated to the PMU. In addition to the PCI-CLK3 signal231, the ethernet controllers unit 233 is coupled to the PCI-bus 217 andthereby to other units within the system that also attach to PCI-bus217, such as for example the CPU 201, South Bridge 223, and VideoSubsystem 236.

A clock signal 228 is also generated and communicated to the input ofPMU 224. These different outputs of clock generator unit 232 arenominally at a frequency suitable for switching circuits-within theirrespective destination units. Clock generator 232 and micro-controller250 also each receive a signal 256 which is operative in the clockgenerator unit and in the micro-controller 250.

Video subsystem 236 is coupled to PCI-Bus 217 and receives PCI-CLK2signal 230 from clock generators 232 and operating voltage VCC(nominally at 3.3 volts) as already described. Video Sub-system 236 isresponsible for generating a video signal 243 and outputting or makingthe video signal available for communication elsewhere or to a display(not shown), such as the display on the front panel 246 of rack or baymounting one or more ISSU. Front panel 246 may also optionally includevideo or LCD displays, indicator lights or LEDs, and the like fordisplaying status or other information or data. Keyboard (K) and mouse(M) (or other pointing device or selection mechanism) may also beprovided and brought out to the front panel (or other location) foraccess.

Voltage regulator Unit 241 receives nominal ±5 volt and ±12 volt directcurrent voltages from power supply module or modules 110 over abackplane connector 103. Each of the expansion or secondary RAM 221,Fast Ethernet Controllers 233, clock generators unit 232, South Bridge223, Micro-controller 250, and video sub-system 236 receive an operatingvoltage VCC 222 nominally at 3.3 volts in this particular embodiment(and a corresponding operating current). While the same VCC is used inthis particular embodiment, it will be understood that the supply or VCCvoltage is selected to match the operating characteristics of the deviceor circuits and that each circuit or device need not operate at the samevoltage. It will also be appreciated that the voltage for one or more ofthese units may be reduced, sequenced (ON/OFF), and/or turned OFFentirely using appropriate power control and in some cases protectioncircuitry. The supply voltage or VCC provided to the CPU 201 in theinventive ISS is controlled or modulated in a particular manner toachieve significant power consumption reduction. These circuits andmethods which utilize one or both of CPU clock frequency control and CPUcore voltage control are described in greater detail elsewhere in thisdescription.

In one embodiment of the inventive server module, on-board voltageregulators 241 generate a nominal regulated 5.0 volt output voltagesignal (VCC_(—)5.0V) 242 for hard disc drive 239, a nominal regulated3.3 volt output voltage signal (VCC_(—)3.3V) 222 which is communicatedto primary RAM 208, expansion or secondary RAM 221 (when present), FastEthernet Controllers 233, clock generators unit 232, South Bridge 223,Micro-controller 250, and video sub-system 236. Primary RAM 208 isselected as DDR SDRAM to advantageously operate at a lower voltage of2.5 volts and receive a separate supply input signal (VCC_(—)2.5V) fromvoltage regulators 241. In an alternative embodiment, the primary RAMmay operate at a somewhat higher voltage (such as 3.3 volt), but in suchsituation the power conservation achieved by the ISS will not be asgreat. Secondary or expansion RAM may also alternatively be selected tooperate at a lower voltage (such as 2.5 volts) to achieve somewhatgreater power consumption savings when present. It is noted that whilecertain voltage ranges, such as 2.5 volt, 3.3 volt, and 5.0 volt arerecited for particular circuits, these voltages are merely exemplary andshould not limit the scope of the invention. It will also be appreciatedby those workers having ordinary skill in the art in light of thedescription provided herein, that certain voltage relationships and thesequencing, modulating, or other control of voltages and/or currentprovides significant power conservation as compared to conventionalsystems, devices, and techniques.

In alternative embodiments of the invention, the operating voltage (VCC)to each unit or to selected groups of units may be supplied viadifferent wires, traces, or busses, so that the on/off condition and/orvoltage level to each device or group of devices may be controlledseparately. Control logic may also be provided within each unit so thateven where a common wire, trace, or bus couples an operating voltage tothe unit, control signals communicated to the logic may exert furthercontrol to remove voltage from the unit or to sequence the voltage onand off according to some predetermined or programmable rules, and/or toreduce the voltage level.

Hard disc (HD) drive 239 is coupled to the Southbridge unit 223 viaconventional means, in this particular embodiment, via an IDE connection240. In the chipset community, Northbridge refers to the major buscontroller circuitry, like the memory, cache, and PCI controllers. Thenorth bridge may have more than one discrete chip. In analogous manner,Southbridge refers to the peripheral and non-essential controllers, likeEIDE and serial port controllers. As used here the Southbridge Unitrefers to the unit that has these type of functionality as it couplesthe PCI-Bus 217 with the X-Bus 238 to the System Flash ROM, and to theIDE (or EIDE) bus 240 that couples the Hard Disc drive 239. Thestructure and functions of Southbridge units, typically implemented as asingle chip, are known in the art.

System Flash ROM 237 (such as may be used for storage of a BasicInput/Output System (BIOS) and CMS is coupled to South Bridge Unit 223via a bus 238, such as for example by an X-bus 238.

In this light, one notes from the server module design of FIG. 11, thatCPU 201 receives a processor supply voltage (VCC CPU) signal 219 as aninput supplied by voltage regulators 241, and that voltage regulators241 receive a processor supply voltage control signal (VCC_CPU_ctrl) 218from the CPU.

Micro-controller is coupled to and receives a slot-ID signal 252 and asignal 245 over Amphus Magic Peripheral Control (AMPC) bus 251. TheSlot-ID signal 252 identifies the slot and is used by themicro-controller 250. Micro-controller 250 is also adapted to receive asignal 253 over a serial connection from the South Bridge unit 223.Micro-controller 250 may also generate a reset signal 254.

Advantageously, the CPU supply voltage (or CPU core voltage) is notfixed or constant, but is adjustable either continuously or in a finitenumber of defined increments so as to maintain CPU device (e.g.transistor) switching speed at a particular clock frequency whileminimizing power consumption by the CPU. In a particular embodimentusing one of the available Transmeta Corporation processors (forexample, the Model TM3200, Model TM5400, or Model TM5600 Crusoeprocessors made by Transmeta of Sunnyvale, Calif.), this voltage iscontrollable from a maximum voltage and maximum CPU/processor clockfrequency to a minimum voltage and minimum frequency. In one embodiment,the maximum voltage is about 1.6 volts at a clock frequency of about 530MHz to a minimum voltage of 1.2 volts at a frequency of 300 MHz.Operation at somewhat higher and lower voltages and/or at somewhatfaster clock rates is likely possible for at least some group ofdevices, but are not within the published operating range of thisprocessor chip.

Operating power and clock switching frequency to the CPU 201, primaryRAM, expansion RAM (when present), Video Subsystem 236, Fast EthernetController 233, South Bridge Unit 223, and Hard Disc drive (HD) 239 inone embodiment are controlled as indicated in Table I. In this powermanagement and control scheme, the first mode (first state) is enteredwhen the CPU is running at maximum frequency. Maximum frequencytypically requires maximum voltage.

The second mode (second state) is implemented by using Transmeta LongRunpower management scheme. This LongRun™ power management scheme issummarized elsewhere in this description as well as in the publicationLongRun™ Power Management—Dynamic Power Management for CrusoeProcessors, Marc Fleischmann, Transmeta Corporation, 17 Jan. 2001;herein incorporated by reference.

The third mode (third state) is entered when activity level drops belowa certain threshold. This activity can be either monitored by anexternal supervisor, or monitored by the Server Module (SM) itself andthen reports the result to the external supervisor. The server module(also referred to as a computer node) will either suspend itself afterreceiving instruction from the external supervisor or after the PMUinside the SouthBridge or other core logic receives a suspend signalfrom the microcontroller.

Aspects of an embodiment of the manager or management module are nowdescribed. As its name implies, the Management Module is responsible forperforming a variety of management or supervisory functions. Some ofthese functions are highlighted immediately below, others are describedrelative to other aspects of the invention throughout the specification.

The Management Module can connect either serial port on any of theServer Modules or the serial port on the Management Module to the backpanel DB9 connector. In one embodiment, this is a full eight-signalswitch (not only TxD and RxD). The Terminal Server portion of theManagement Module provides Ethernet connection services for all of theServer Modules' serial ports.

FIG. 12 is a diagrammatic illustration showing the relationship andconnectivity between a plurality of server modules 112, two managementmodules 108, and two Ethernet switch modules 104. In this particularembodiment, the lines represent the differential pair signals that areused for signaling.

The Management Module is also responsible for Ethernet switchmanagement. Management, control, and status reporting of any presentSwitch Modules are performed via the Management Module. Industrystandard SNMP MIBs (Management Information Bus) are advantageouslyimplemented providing support for existing data center management toolsand is under security protection. Other protocols may alternatively beused.

Embodiments of the Virgo ISSU contains dual redundant Fan Modulesconsisting of three fans each. The Management Module is responsible forFan Control and Monitoring. Each fan's speed is desirably controlled toreduce system noise and extend fan life, while maintaining internaltemperatures at appropriate levels. When an excessive temperaturereading or fan failure is detected, alerts are sent via email or SNMP,depending on system configuration.

The Management Module is also responsible for Power Supply Control andMonitoring. The power supplies of the Server and Switch Modules arecontrolled by the Management Module, with the ability to cycle power orshut down a module upon receipt of an appropriate network control centermessage and is under security protection. All power supplies, eitherchassis or module based, are monitored. When an alarm or failurecondition is detected, an alert is sent via email or SNMP depending onsystem configuration.

The Management Module provides out of band reset capability (resetcontrol) for Server and Switch Modules. The mechanism used for resetdepends on system configuration and is advantageously under securityprotection.

Embodiments of the Management Module provide multiple, for examplethree, 10/100 Full Duplex Ethernet Ports that provide redundantcommunications to internal and external networks. There is an EthernetPort on the Rear Panel. There is also an Ethernet Port internal to theVirgo ISSU System Chassis to each of the Switch Modules. The ManagementModule can be configured and Terminal Service provided via all of thesepaths.

When a Virgo ISSU System contains two Management Modules the moduleswill cooperate in a load-sharing configuration. This is referred to asRedundant and Shared ISSU System Control. Both Management Modules willcontinuously monitor and report the status of the system and allsub-systems. Should one Management Module fail, or not be installed, theremaining Management Module is capable of controlling the entire VirgoSystem.

Furthermore, the Management Module(s) are responsible for general systemhealth monitoring and it (they) constantly monitor the health of theVirgo ISSU-based system. The system's health is monitored bycontinuously checking temperatures throughout the chassis. All chassispower supplies and all power supplies on Server, Switch, or ManagementModules are checked via the Management Module. Each of the six fans inthe system is monitored for low speed and failure to operate. When analert or warning condition is observed, notification is sent to adesignated recipient via either email or SNMP. It is possible to setwarning levels as well as the notification method via systemconfiguration.

Embodiments of the Management Module 108 is designed to be “plugged-in”and operate within 2U rack mountable Chassis OR Enclosure101. Thecombination of a Management Module 108 along with one or more ServerModules 112 and Switch Modules 104, constitutes a complete ISS system.One module of each type is sufficient to form a complete system.

The Integrated Server Systems (ISS) architecture and ISSU system unitsare designed to offer the lowest power consumption, the highest density(number of servers per rack) and highest MASS (Manageability,Availability, Scalability, and Serviceability). It offers data centeroperators a breakthrough platform that reduces TCO (total cost ofownership) and increases revenue per square foot through dramaticallyimproved performance efficiency (e.g. transactions per second per watt)as compared to conventional architectures and system configurations.

As data centers grow in size and complexity, it becomes increasinglydifficult to manage server and networking resources. The ManagementModule 108 is designed to consolidate the monitoring and management ofServer Modules 112 (up to 16), Switch Modules 104 (up to 2) and otherhardware subsystems enclosed in a single system 102. This reducesmanagement complexity, and provides a more cost-effective solution thanexisting management servers.

In at least one embodiment, a management module provides numerousfeatures and capabilities, including: Local Serial Port Switching and/Remote Terminal Server, Ethernet Switch Management, Fan Control andMonitoring, Power Supply Control and Monitoring, In-band and Out-of-BandCapability, Remote Warm Boot and Cold Boot Control, Dual ManagementModules for High Availability Management, and Browser-based access forConfiguration and Management Controls, as well as KVM over the ethernet.

Local Serial Port Switching and /Remote Terminal Server features areprovided in that the Management Module can switch any of the serialports on the Server Modules 112, or the serial port on the ManagementModule 108 itself, to a DB9 or other connector at the rear of thesystem. This provides serial console output capability for softwareinstallation, out-of-band communication, configuration and management.Additionally, the Management Module can redirect the serial input andoutput of any selected port over Ethernet for Remote Terminal Servercapability.

Ethernet Switch Management is provided in that the Management Module 108performs management, control, and status reporting of Switch Modules104. In one embodiment, industry standard SNMP MIBs are implemented onthe Management Module 108 providing support for existing SNMP managementtools such as OpenView and Unicenter. This capability may advantageouslybe via-secure access.

Fan Control and Monitoring is provided in that each ISSU contains dualFan Modules 114 consisting of three fans each 114 a-1, 114 a-2, 114 a-3.Each fan's speed is controlled to reduce system noise and extend fanlife, while maintaining internal temperatures at targeted andadvantageously at optimum temperature levels. When an excessivetemperature reading or fan failure is detected an alert is sent viaemail or SNMP, depending on system configuration so that correctiveaction may be taken automatically or with the intervention of a humanoperator.

Power Supply Control and Monitoring features are implemented in that thepower supplies 110 a-1, 110 a-2 of the Server Modules 112 and SwitchModules 104 are controlled by the Management Module 108, with theability to cycle power (or voltage) or shut down a module upon receiptof an appropriate message from the Information Technology (IT)administrator's network control station. This feature may be via secureaccess. All power supplies, whether a part of the system chassis 101,Server Module 112, or Switch Modules 104, are monitored for proper powersupply operation. When an alarm or failure condition is detected for anypower supply module 110 an alert or other information bearing message issent via email or SNMP, depending on system configuration.

In-band and Out-of-Band Capability is provided in that the ManagementModule has three 10/100 Full Duplex Ethernet Ports that provideredundant communications to internal and external networks. There is anEthernet Port on the rear panel of the system 102 that can be used forout-of-band Ethernet communication. The other two ports connect to thetwo internal Switch Modules (if both are present) 104 a-1, 104 a-2 forin-band communication. The Management Module 108 can be configured andTerminal Service.

The Remote Warm Boot and Cold Boot Control feature of the ManagementModule 108 provides remote reset and power cycling capabilities forServer Modules 112 and Switch Modules 104. Such remote warm boot andcold boot control can be performed in-band, through the Switch Modules104, or out-of-band via the Management Module 108 external Ethernetport. The mechanism used depends on the system configuration and may bevia secure access.

The (optional) presence of two or dual Management Modules support HighAvailability Management. A system can accommodate up to two ManagementModules 108 for high availability. For alternative physicalconfigurations, additional management modules may be provided for evengreater availability, though such redundancy would seldom if ever beneeded. When both management modules are present in a system, theyassume an “active-active” load-sharing configuration. Both ManagementModules 108 a-1, 108 a-2 continuously monitor and report the status ofthe system and all sub-systems. Should one Management Module fail, ornot be installed, the remaining Management Module is capable ofcontrolling the entire system.

The system also supports browser-based access for Configuration andManagement Controls. Standard SNMP management software is used in oneembodiment to not only monitor and manage the rest of the system 102through the Management Module 108, but also to control (self-control)the Management Module itself. Similar capabilities are alsoadvantageously provided via a browser-based interface.

Attention is now directed to a description of a network and managementof elements of the network by two different types of master relative toFIG. 13. Two different types of master module, M1 501 and M2 502, aredepicted in FIG. 13. Note that the two different types may be configuredwithin a single hardware/software management module and that their typecharacteristics may be defined by a software or program switch. The type1 master M1 501 has the capability to power manage the servers 510within the same subnet 504 using either in-band 506 or out-of-band 508signals. Furthermore, the type 1 master 501 can also power manage theservers (e.g. Web Servers 512, 518, Cache Server 514, 520, and StreamingMedia Server 516) outside of its own subnet 504, for example in outsideor external subnet 522, 524. Either type 1 or type 2 master (M1 501 orM2 502) can be designated as the global master used to manage the restof the servers throughout the entire internetwork 530. Any server hasthe option to power manage itself as well as being managed by the globalmaster. The global master communicates with all servers via standardnetwork management protocol. Each server has its own network agent andwill report (immediately or within some predetermined time interval) tothe global master on any policy violation.

An exemplary operational scenario for the internetwork is now described.Assume for example, that while a particular server is operating in the2nd power mode and the network agent detects the CPU utilization forthat server rises above an upper threshold (for example, a threshold ofabout 95%) for some fixed period of time, this is considered as a policyviolation and a message will be sent to the global master. Consequently,the global master will command the server to return to the 1st mode,that is a higher power consuming and higher performance mode since itmay be anticipated that with only a 5% excess capacity margin remainingunused, quality of service may suffer if there is an additional loading.This task may alternatively be performed by the local CPU.

On the other hand, while operating in the 1st mode, if the agent detectsthe CPU utilization for that server module drops below a lower threshold(for example a threshold of about 50%) for a fixed period of time, itwill send a policy violation signal to the global master. The globalmaster will command the server to enter the 2nd mode wherein powerconsumption is reduced relative to the 1st mode, such as by reducing theCPU clock frequency, lowering the CPU operating voltage, and preferablyboth lowering the CPU clock frequency and the CPU core voltage to matchthe switching speed requirements imposed by the clock frequency. Thistask may also alternatively be performed by the local CPU.

If the global master observes that the CPU utilization of multipleservers continue to stay at very low level, it will command some of theservers to enter the 3rd mode in order to save the maximum power. The3rd mode is a power consumption mode that is even lower than the 2ndmode. In one embodiment, the CPU clock is stopped, while in anotherembodiment, the CPU is reduced to a very low rate. Advantageously, theCPU core voltage is also reduced but this is not required. Preferably,the CPU core voltage is not reduced below a voltage level that willresult in the loss of state within the CPU. That is the CPU is desirablynot powered off.

In a typical server farm, it is anticipated that the nominal load isabout 30% of the peak load. Using this approach, the inventive structureand method can provide at least a 70% power saving comparing with serverfarm that does not implement the inventive power management. Some of thefigures show these transitions in diagrammatic form.

A graphical plot of CPU Utilization (in percent) versus Time isillustrated in FIG. 14. It is assumed that at time zero CPU utilizationfor the particular server module operating in 2nd mode is 50% and thatthis rises toward 100%. When the utilization reaches about 95% and staysover that level for some predetermined time, control signals aregenerated and received by the server module in order to transition theserver module from the 2nd mode to a higher load capacity 1st mode. CPUloading as a percentage of available capacity immediately drops due tothe higher capacity while operating in 1st mode. While operating in this1st mode, utilization drops below 50% but then immediately arises above50%. Because the utilization did not remain below 50% for the prescribedperiod of time, no transition to 2nd mode occurred. However, at asomewhat later time, CPU utilization dropped below 50% for the requiredtime and resulted in a transition to the 2nd mode. As utilizationcontinued to decrease and remain at that reduced level, a furthertransition to the 3rd mode occurred. In this exemplary embodiment, the3rd mode is an inactive mode where the CPU is inoperative thereforethere is zero utilization.

Though not shown in the diagram, the particular CPU and servercontaining it may remain in this 3rd mode until its capacity is requiredfor the system as a whole. At such time, control signals are provided tothe particular server, as described elsewhere in this description, toactivate the CPU into an appropriate performance state. This state maybe either a 1st mode or a 2nd mode, depending on the loading andpolicies in place.

It is noted that although the 95% and 50% CPU utilization thresholds,and certain relative periods of time may be inferred from thedescription or drawings, the CPU utilization control and powertransition policies may be freely defined to satisfy the dynamics of thesystem and the conservative or liberal achievement of power conservationwithin the system. For example, the threshold for transitioning from a2nd mode to a 1st mode may be selected to be any CPU utilization between60% and 100%, though more typically it will be selected to be betweenabout 80% and 95%, and more frequently between about 90% and 95%. Inanalogous manner, the threshold for transitioning between the first modeand the second mode may typically be selected to be a utilizationbetween any predetermined limits, such as for example between about 25%and 55%. The system may also be structured to that the threshold orthresholds are set or adjusted dynamically. Dynamic thresholding may forexample be based at least in part on present, past, and/or predictedfuture loading, Quality of Service indicators or factors, or othermeasures or parameters.

Furthermore, while a simple utilization threshold has been described,more elaborate control may be implemented. For example, in addition tothe threshold, a velocity measure of the time rate of change ofutilization may be utilized as well. This may then result in adynamically determined threshold that depends upon how fast theutilization is changing and its direction. Ultimately, the goal of thepolicy should be to anticipate the need for additional CPU capacitybefore capacity is exceeded in one direction, and to ramp down powerconsumption without reducing CPU capacity below a level that can satisfyshort term demand in the other direction.

Another function of the global master is to reconfigure the server type(such as between a web server type, a cache server type, a streamingmedia server type, or other type server) based on activity monitoring atthe application level. Alternatively speaking, if 70% of the servers arebeing held in a suspend state, the global master should have the optionto turn those servers into whatever types that are in demand at thetime. The global master may even decide that a particular server mayserve for more than one type of content simultaneously, that is operateas both a web server and a cache server. This may be particularlyappropriate when the loading on all servers is relatively low so that asingle server is capable of satisfying quality of service requirementswithin its CPU (and hard disk drive) capacity.

Having now described the physical architecture and connectivity of anexemplary Integrated Server System, the structure and operation of anexemplary server module, management module, and switch module, aspectsof how these modules work independently and in synergistic manner toaccomplish significant power or energy conservation without sacrificingperformance (or with only an undetectable insignificant performanceimpact at most) are now described.

Conventional servers do not take power consumption or power savings intoconsideration in their normal operation. The primary philosophy of datacenter and internet service providers (ISPs) is over provision. If oneconsiders the relationship between traffic (or load) and the powerconsumption, conventional servers operate at a relatively constant andhigh power level that is independent of load. The processors andassociated memory typically run at maximum clock rate or frequency, fullsupply voltage to the processor core, hard disc drives on and rotatingconstantly, and video and other sub-systems and ports on and operableall the time independent of whether they are being used at that time.

Power conservation features such as may be provided in consumer personalcomputers (PCs) such as the suspend, sleep, hibernation, and the liketypes of reduced power operation. Several different power managementinterface specifications have been developed in recent years, includingfor example, the Advanced Configuration and Power Interface Version 1.0(herein incorporated by reference) co-developed by Intel, Microsoft andToshiba, which specifies how software and hardware components like theoperating system, motherboard and peripheral devices (such as hard diskdrive) talk to each other about power usage.

One embodiment of the inventive structure is directed as so called“front end server” applications. While the power consumption ofconventional servers may vary depending upon the processor type andoperating characteristics, number of processors, amount of memory (RAM),disc drive or other storage device type and number, and the like, mostconventional servers such as those made by Cobalt, Compaq, Dell, andothers consume some where in the range of between about 50 Watts to 150Watts or more. Some servers have as many as four processors and willconsume considerably power.

Conventional servers typically have similar architecture to personalcomputers made for home and business use, albeit with greater I/Ocapabilities and horsepower for their intended tasks. Interestingly,most of these servers retain the video capabilities in spite of the factthat the servers will not be used by anyone for viewing the video. It isunfortunate that the video circuitry (either within the processor or asa separate chip) is consuming power yet produces no beneficial effect orresult.

The structures and methods of the invention provides a very low powerdesign so that even when the inventive server is operating at itsmaximum performance level and consuming its maximum power, that maximumpower consumption is still a fraction of the maximum (and steady-state)power consumption of conventional non-power managed processors andservers. This maximum power level is typically between about 10 to 15Watts though it may fall within other ranges or be reduced further. Thisreduction is possible for several reasons, including the provision of avery low power consumption processor or CPU, turning off devices orcomponents within the system that are not being used at the time.Another significant power savings is provided by power managing the CPUaccording to the network traffic or server load conditions. Thereforethe power consumption is less than the maximum power consumption unlessthe load is at a peak and all of the devices and components are poweredon to handle the load. With this throttling back as a function of load,the power consumption may be at any intermediate value between zero(when and if the unit is powered off completely) or at a very low powerconsumption level when placed in some power conserving mode (such as asleep, suspend, or other specialized power conserving mode as describedelsewhere herein). Thus, capabilities of the server are matched to thedemands being placed on the server. This power control or management isreferred to as power on demand (Power on Demand™) and permits powerconservation without any loss of server capability. Power management mayalso be controlled dynamically.

The over-provisioning of servers by ISPs and Data Centers is adapted atleast in part because e-commerce can be highly seasonal and subject toconsiderable event driven demand surges. For example, the traffic orload requirements placed on servers during Christmas Holiday season maybe many time or even one or more orders of magnitude as compared toother times of the year. News, stock market, and other organizations mayhave analogous traffic fluctuations during a single day. Unless suche-commerce entities are able to satisfy the inquiries of their customerswith tolerable quality of service (QOS), such customers may never comeback to the site. Therefore, day-to-day, week-to-week, andmonth-to-month traffic loading can vary over a wide range. For onetypical ISP, the average load is about twenty-percent (20%) of themaximum load.

In the inventive system and method, by varying the power consumptionaccording to load, considerable additional savings are realized. For anexemplary system in which the base maximum power consumption is 10 wattsrather than 50 watts, and the power consumed during the service cycle ison average 20% of the maximum, the net result is a realization of theproduct of these two savings for a savings of about 25 times. That isthe power consumed over a day is 1/25 of the power consumed for aconventional server operation.

Typically, the amount of power savings and then relationship betweentraffic and power consumed will depend upon the nature of the server.For example, a web server may exhibit a different load versus powerconsumption characteristic curve than a streaming video server, whichwill be different that a content caching server. These relationships maybe linear or non-linear. The nature of the content may also impact thisrelationship.

The inventive scheme interactively reacts to the load and scales thenumber of components and/or devices as well as the operationalparameters and operating characteristics of the devices and/orcomponents to match the load or a predetermined quality of service, orsome other identified performance target for the server system.

The inventive ISS may incorporate multiple servers adapted to servedifferent type of content. Thus it may be expected that each differentserver will exhibit somewhat different power consumption and powerconsumption reduction characteristics. These characteristics need not beknown a priori to realize their benefits.

Attention is now directed toward a description of exemplary differentoperating modes. In one aspect the inventive structure and methodprovide for a transition in a single processor or CPU between a firstmode (Mode 1) and a second mode (Mode 2) wherein the second modeconsumes less power or energy than the first mode. Power or energyconsumption in the processor or CPU (and optionally in other circuitcomponents or peripherals connected to or associated with the processoror CPU) may be reduced in a variety or ways, including for example,lowering a processor or CPU core voltage, reducing a processor or CPUclock frequency, or lowering the core voltage and the clock frequency atthe same time.

In some systems and methods, the core voltage and clock frequency arechanged continuously or in stages in some synchronized manner, as ahigher core voltage may typically be required to support a fasterprocessor or CPU clock frequency. It is noted that the first and secondmode are each active operating modes in which the processor or CPU isexecuting instructions and carrying out normal processor functions.While the core voltage may be reduced, the processor clock is stillcycling at some nominal rate. The lower limit for processor clockfrequency reduction may generally be selected based on the types ofprocessing that may be accomplished at that rate. For example, firstmode operation would typically be provided at substantially 100% of thenominal rated clock frequency for the processor, while second modeoperation provide a clock frequency less than 100%. Such reducedprocessor clock frequency may generally be in the range of between about5% to 95% of the maximum, more usually between about 20% and about 80%,more usually between about 20% and 60%. In some systems, the processorclock may be reduced by factors of two using clock signal divisioncircuitry. In other systems, the processor clock frequency may bereduced in fixed increments or according to a clock frequency ratereduction look up table or algorithm in a clock generator circuit. Asthe second mode may be considered to be any active operating mode lessthan the first mode, it will be understood that there may be multiplelevels of this second mode. That is, Mode 2 may be multi-level.

In addition to these first and second modes, the processor or CPU may beplaced into an inactive third mode (Mode 3) characterized by consumingless power or energy (conserving more power or energy) than in the firstmode or the second mode. This third mode is referred to as an inactivemode as the processor clock will be stopped or operate at such a lowfrequency that the processor effectively processes no instructions andperforms substantially no useful work relative to the amount of workprovided in the first or second modes. Usually, the processor clock willbe stopped and where core voltage control is available, the processorcore voltage will be reduced to a level just sufficient to maintainprocessor state. This third mode is distinguished from a fourth mode(Mode 4) where the processor is powered off and does not maintainprocessor state, revitalization of the processor from the fourth moderequiring a reboot or other initialization procedure. Such reboot orinitialization procedures typically requiring a few to tens of secondsto accomplish and compared to fractions of a second to transition theprocessor from the third mode to the second mode or to the first mode.

The present invention provides and supports several differentstructures, mechanisms, and procedures for controlling the operationalmodes of the server modules and hence the processor or processors thatmay form or contribute to the operation of a server. Organizationally,the control may reside in a separate Management Module, one or two ofwhich Management Modules may be integrated into one of the inventiveISSU; or, may reside in one of the Server Modules which has beendesignated as a manager, supervisor, or master server module.Designation of a Server Module in this way involves providing the servermodule with the computer program software for receiving activityinformation from the server modules, for analyzing the activityinformation to determine from a power consumption perspective (or otherpredetermined perspective) which server modules should be operated inthe several available modes (for example, Mode 1, Mode 2, Mode 3; andMode 4 in some circumstances), and where the operation of more than onetype of server is to be combined into a single server module (such as amulti-media server and a web page server) for gathering the content fromthe type types of servers onto the hard disk drive of a single server orgroup of servers. Note that when a particular server module is to serveas the master, that server may collect information on its own activityand be considered in the overall server and power management scheme. Ofcourse, the server module acting as its own master will not typicallyplace itself in either Mode 3 or Mode 4 as its continued operation isnecessary to control other server modules. Where appropriate logic isprovided to place the master in a power conserved state (such as Mode 3)and bring it out of that state, even the master may be placed into oneof the Mode 3 operating states.

At a top level, the server modules each detect and report their activityto the manager (either the management module or the designated masterserver module). In some embodiments, the server modules are permitted tolocally control their own operating mode, for example whether their ownCPU (or CPUs if a multiple CPU server) is or should'be operating in aMode 1 or Mode 2. They will then also report not only their activitylevel but also the operating mode under which the reported activity wasmeasured or detected.

At another level, the manner in which activity is detected is an issue.At yet still another level, the power management control policy orprocedure, that is the control plan that regulates which server modulesshould be place in which of the available modes to provide both therequired (or desired) performance according to some measure and therequired (or desired) power conservation. Those workers having ordinaryskill in the art will appreciate, in light of the description providedhere, that there are virtually limitless different policies for powermanagement. Specific policies that optimize or near-optimize thecombination of server performance and power conservation may bedetermined empirically during initial installation and operationalphases as they will likely depend upon the content served, the variationof server loading as a function of time of day, advertising orpromotions, average server loading, amount of over-provisioning, minimumquality of service requirements, power consumption of server modulesversus content served, and other factors associated with serveroperation. The policies may also be modified according to the particularphysical and/or electronic or logical structure of the servers. Evendifferent CPU technologies may suggest different policies.

It may also be observed that such policies may be biased in favor of anyone or combination of server operational factors. For example, operationand therefore the policy for control may favor power conservation eventhough there may be some impact on performance. Alternatively, thepolicy may favor absolutely maintaining a quality of service even ifpower conservation is somewhat sacrificed.

As general multi-power management policy it is observed based onanalytical and empirical data, that there is a certain power consumptionoverhead associated with each server device and that it is thereforegenerally preferred to operate a minimum number of server modules atnear their maximum output (Mode 1). When a single device approaches itscapacity, other server devices are brought up from a Mode 3 to Mode 2 orMode 1 operation. Frequently, the two servers then on line would eachoperate in Mode 2 until further performance is needed, at which time onewould be brought to Mode 1 operation. This is merely an example scenarioand many other alternative control strategies may be applied. Clearly,there is a bodies of knowledge for both open-loop and feed-back basedcontrol that may be used by those skilled in the art to optimize ornear-optimize some weighted combination of performance and powerconservation.

A server system configuration tool may be provided that allows a humanoperator to monitor system operation and power consumption and interactwith the system and policy definition within the system to tune systemperformance. In the event that local government or regulatory agenciesrestrict power consumption or mandate power reduction, the policy may bealtered to implement these requirements. In each of these situation, thesystem permits real-time dynamic uploading of the policies withouttaking an servers offline. In one embodiment, systems having twomanagement modules are used effectively by off loading one managementmodule to the other management module, updating the policies in the offloaded management module, and then placing the updated managementmodule. In another embodiment, alternative policy schemes are preloadedin the management module (or designated master) so that it may switchautomatically or under operator control as required.

In one embodiment of the invention, the computer system comprises aserver for serving data or other content in response to a request. Ahypothetical scenario in which a computer system, which may typically bebut not necessarily be a portion of a larger network system havingmultiple server computers, transitions from a full power maximumperformance operating mode to an off state in which the computer systemneither performs operations no maintains state. The particularprogression between states or modes may possibly but is unlikely tooccur in a real computer system as it is more likely that certain modeswill be skipped either to reduce power consumption when performancerequirements are low or skipped when performance demand increases so asto elicit a higher performance operating mode than the next progressionwould provide. In general, the inventive system and method may providefor transitioning between an one state and any other different state. Insome embodiments of the inventive system and method, not all of themodes described here will be present. Furthermore, other embodiments ofthe invention may provide for additional and different control.Furthermore, the description immediately below addresses control of theprocessor unit (e.g. processor or CPU) and logic circuits (frequentlyreferred to as core logic or SouthBridge) associated with the processorunit. It should be understood that control of other components withinthe system, including for example hard disk drives, input/output ports,network interconnect circuits or cards, BIOS, video circuits, clockgenerators, voltage regulators, micro-controllers, memory, as well asother individualized logic circuit components may be independently ordependently controlled or controlled as groups. (See for example, TableIII and the accompanying description for the manner in which someelements are controlled.)

It is initially assumed that the system is operating in Mode 1 havingthe highest processor unit (e.g. CPU) performance level and greatestpower consumption of the available operating modes. The system isconfigured with operating system software (e.g. Microsoft Windows,Linux, Unix, Sun, or the like) and/or applications program software thatinclude instructions for monitoring the occurrence or non-occurrence ofan event.

It is noted that the Linux Operating system, such as the RedHat Linuxoperating system, may be more power conserving than other currentlyavailable operating systems. One reason for its power conservativefeatures are the fewer number of instructions that need to be executedto accomplish tasks. Therefore while embodiments of the inventionsupport all of the available operating systems, and may be adopted tosupport future operating systems, one embodiment utilizes the Linuxoperating system to achieve a higher degree of power conservation.

One such event that can be monitored and detected is the occurrence ofexecution of an idle thread. Another such event is the occurrence ofsome specified level of CPU processing capability availability that isderived from some enumeration or statistical evaluation of the idlethread or idle threads that are being or have been executed during sometime period. Other events that may trigger a transition are describedelsewhere in this specification. For purposes of continuing thedescription here, it is assumed that execution of idle threads ismonitored and reported by a combination of an application program andthe operating system, and that the number of idle threads being executedsuggests that more performance is available than is needed and thatpower consumption may be reduced without sacrificing performance.

Control signals are then generated (either locally by the CPU or corelogic, or globally by a separate power manager) that transition thesystem from Mode 1 to one of the Mode 2 operating modes. Mode 2 isgenerally characterized by having a CPU clock frequency that is lessthan the maximum rated CPU clock frequency, a CPU core voltage less thanor equal to the rated maximum CPU core voltage, and core logic thatoperates at or substantially at the rated core logic clock frequency andcore logic operating voltage. (This condition is also referred to as theMode 2′ operating mode.) By maximum rated CPU clock frequency isalternatively meant: (i) the clock frequency the CPU manufacturer hasidentified with this CPU model, (ii) the actual maximum frequency atwhich the CPU may be clocked, (iii) the maximum clock frequency that theCPU is operated within the system independent of what the CPU is capableof being operated at, (iv) or some similar or analogous measure. Forexample, if the CPU is marketed or sold as a 800 MHz Intel Pentium III,then the maximum rated CPU clock frequency is 800 MHz. If the maximumclock frequency at which the 800 MHz Intel Pentium III is operated inthe system is 850 MHz, then the maximum rated frequency is 850 MHz.

It is also understood that there are gradations of performance (andpower consumption) within the rubric of Mode 2 operation. A Mode 2″operating mode is characterized by operation at both less than themaximum rated clock frequency and at less than the maximum rated corevoltage. Mode 2 may be a single operating mode, or include a pluralityof operating modes, having the general Mode 2 characteristic butproviding for several different CPU clock frequencies and core voltagethat at least support electrical device switching (transistor switching)or be selected to provide just adequate core voltage substantiallymatched to the clock frequency to provide reliable operation. Forexample, at the Mode 2″″ operating mode, the CPU clock frequency and CPUcore voltage are the minimum clock frequency and core voltage that areoperable and supported by the CPU (where such minimum exists).Embodiments of the inventive system typically provide that core logiccontinue to operate at nominal rated levels where both the core logicclock frequency and core logic operating voltage are at or substantiallyat rated levels. In other embodiments, of the invention core logiccircuit elements may also be power managed during Mode 2 operation byreducing clock frequency, operating voltage, or both.

The CPU clock frequency may be adjusted over a range of frequencies tomatch the amount of processing capacity to the tasks to be performed.Therefore, as the number of idle threads being executed in the CPUcontinue to increase indicating that productive tasks (such asretrieving data from a storage device, and sending such retrieved datato an I/O port or NIC for serving to a requestor) are being performedwithin specified limits or some quality of service parameters, the clockfrequency may be continually reduced.

At some time, however, the requirements placed on the system may becomesome low that at times there are no tasks to be performed. For example,on a computer network having a multiplicity of servers for serving stockmarket quotes and having sufficient capacity to handle worst casetraffic in an active stock market, there is likely to be lots of overcapacity of a national holiday where the stock markets are closed andthere is little interest among investors. Under such conditions(actually likely under less strenuous conditions than these) the CPUwithin a computer system may complete all pending applications or usertasks and begin executing a system idle loop. Such an idle loop mayinitially or after some period of time cause execution of a CPU Haltinstruction (or the equivalent) that causes the CPU clock to stop. ThisCPU halt instruction may be generated by the CPU itself or through someother internal or external agent or program. For example, a MicrosoftWindows operating system or a Linux operating system are capable ofgenerating an instruction to halt the CPU or processor. A halted orstopped CPU is one example of a Mode 3 operating mode, and moreparticularly a Mode 3′ operating mode that is nominally characterized bya stopped or substantially stopped CPU clock, and a CPU core voltagethat is less than or equal to the nominal maximum CPU core voltage andmore usually at the minimum CPU core voltage that is necessary tomaintain CPU register state and/or other CPU state. A CPU suspend stateis another example of a different type of Mode 3 operation. Mode 3″ mayrepresent further power conservation by lowering the CPU core voltage tothat just required to maintain state. This is treated as a separate submode because CPU core voltage need not be reduced as a result of the CPUhalt command, and as stopping the CPU clock for a short period of timebetween execution of application tasks itself provides significant powersavings without the design changes that may be required to alsotransition core voltage. Reduction of core voltage when the clock isstopped also generally has a smaller impact on power conservation thanwhen the CPU is clocking. Some embodiments will also operate the CPU atthe minimum clock frequency and minimum CPU core voltage as providedunder a Mode 2 operation, and when executing the CPU halt instructionturn off the clock from that minimum value and maintain the core voltageat the voltage that supports the minimum clock. In this manner, the CPUmay halted and resumed from halt by restarting the clock and leaving thevoltage alone. This scenario may be particularly effective when makingrapid transitions between Mode 2 and Mode 3.

When it is determined that the CPU and computer system in which theprocessor is installed are not needed for some longer period of time, itis possible to provide additional power savings by reducing the powerconsumed by the core logic circuits or chips associated with the CPU.Where this additional level of power reduction is desired, the corelogic clock frequency may be reduced to something less than the nominalor maximum frequency and in addition but optionally, the core logicvoltage may be reduced so as to support that frequency. CPU and corelogic state are maintained in each of the Mode 3 operating modes.

When the computer system is not needed for some longer period of time,the processor or CPU and at least a substantial portion of the corelogic may be turned off. This is represented by Mode 4 operation whichin one embodiment is characterized by having the CPU core voltage atzero, the CPU clock frequency at zero, most of the core logic circuitsreceiving no operating clocks or operating voltage. In some embodiments,the real-time clock may continue to operate and/or one or more circuitsmay remain active so that they may receive an external signal (such as aWake-on-LAN) derived signal and be turned back on to resume operation inone of Modes 1, 2 or 3.

Note that in some embodiments, wherein if a portion or the entire systemis operating in a reduced power consumption mode, such as one of themode 3 operating modes, the manager or supervisor (such as a managementmodule determines that server modules are dropping packets and that fewor no idle threads are executing (indicating that the system hasinsufficient performance capability) then the supervisor or manager cansend a packet to the fast Ethernet controller (or other circuit)associated with the server module to wake it up. This packet may be anypacket identified to the ethernet controller (or other controller) towake up the server. In one embodiment, the line or bus is monitored forspecial “ON” packet. In another embodiment, any packet received willturn it on. This wake up feature is advantageous as when the processoror CPU is not being clocked (such as in a Mode 3 operating mode)additional means are needed to wake it up to place it in a active modethat can process instructions and perform useful tasks, and in a serverenvironment, the server according to embodiments of the invention willbe connected to the ethernet and active. Other types of wake up orattention signals may alternatively be used.

When performance requirements increase, the computer system maygenerally transition from lower performance (and lower powerconsumption) modes to higher performance (and typically higher powerconsuming modes) according to rules, policies, algorithms, and/orcontrol mechanisms provided in the system. Transitions may also occurdynamically. The events which trigger change or transition from oneoperating mode to another operating mode may typically be set andchanged under programmatic software or firmware control. Variousexemplary situations or events that trigger transitions are describedelsewhere in this specification.

While a number of modes (Mode 1, Mode 2, Mode 3, and Mode 4) have beendescribed in this example, it is noted that the inventive system,method, and computer programs do not require each of these modes or eachof the submodes (e.g. Mode 3″) within a mode. Furthermore, dependingupon the configuration of the system, the set of rules or policies inplace during operation, and/or the dynamics of operation at the time anoperating mode decision is to be made, for any single computer system,or group of computer systems, and their included processor, processingunit, or CPU, operation may transition between any two of the modesdescribed. The examples provided here and the modes or states identifiedin the state diagrams are therefore illustrative rather than limiting.

By way of highlighting selected ones of the computer system (forexample, server module computer system) operating modes, embodiments ofseveral of these modes and submodes are now briefly described.

One embodiment of a first mode (Mode 1) comprises a mode in which theprocessing unit is operated at substantially maximum rated processingunit clock frequency and at substantially maximum rated processing unitcore voltage, and the logic circuit is operated at substantially maximumrated logic circuit clock frequency and at a substantially maximum ratedlogic circuit operating voltage.

One embodiment of a second mode (Mode 2) comprises a mode in which theprocessing unit is operated at less than maximum rated processing unitclock frequency and at less than or equal to a maximum rated processingunit core voltage, and the logic circuit is operated at substantiallymaximum rated logic circuit clock frequency and at a substantiallymaximum rated logic circuit operating voltage.

One embodiment of a second submode (Mode 2′) further comprises a mode inwhich the processing unit is operated at less than maximum ratedprocessing unit clock frequency and at less than a maximum ratedprocessing unit core voltage, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency and at asubstantially maximum rated logic circuit operating voltage.

Another embodiment of the second submode (Mode 2″) further comprises amode in which the processing unit is operated at less than maximum ratedprocessing unit clock frequency and at less than a maximum ratedprocessing unit core voltage, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency and at asubstantially maximum rated logic circuit operating voltage.

Another embodiment of a second submode (Mode 2″′) further comprises amode in which the processing unit is operated at less than maximum ratedprocessing unit clock frequency and at less than a maximum ratedprocessing unit core voltage just sufficient to maintain switchingcircuits in the processor unit at the processing unit clock frequency,and the logic circuit is operated at substantially maximum rated logiccircuit clock frequency and at a substantially maximum rated logiccircuit operating voltage.

One embodiment of a third mode (Mode 3) comprises a mode in which theprocessing unit is operated at a slow but non-zero frequency processingunit clock frequency and at less than or equal to a maximum ratedprocessing unit core voltage sufficient to maintain processor unitstate, and the logic circuit is operated at substantially maximum ratedlogic circuit clock frequency and at a substantially maximum rated logiccircuit operating voltage;

One embodiment of a third submode (Mode 3′) further comprises a mode inwhich the processing unit is operated at a substantially zero frequencyprocessing unit clock frequency (clock stopped) and at less than orequal to a maximum rated processing unit core voltage, and the logiccircuit is operated at substantially maximum rated logic circuit clockfrequency and at a substantially maximum rated logic circuit operatingvoltage;

Another embodiment of a third submode (Mode 3″) further comprises a modein which the processing unit is operated at a substantially zerofrequency processing unit clock frequency (processing unit clockstopped) and at a processing unit core voltage just sufficient tomaintain processor unit state, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency and at asubstantially maximum rated logic circuit operating voltage.

Another embodiment of the third submode (Mode 3″′) further comprises amode in which the processing unit is operated at a substantially zerofrequency processing unit clock frequency (processing unit clockstopped) and at a processing unit core voltage just sufficient tomaintain processor unit state, and the logic circuit is operated at alogic circuit clock frequency less than a maximum rated logic circuitclock frequency and at a logic circuit operating voltage that is lessthan or equal to a maximum rated logic circuit operating voltage.

Another embodiment of a third submode (Mode 3″″) further comprises amode in which the processing unit is operated at a substantially zerofrequency processing unit clock frequency (processing unit clockstopped) and at a processing unit core voltage just sufficient tomaintain processor unit state, and the logic circuit is operated at alogic circuit clock frequency less than a maximum rated logic circuitclock frequency and at a logic circuit operating voltage that is lessthan a maximum rated logic circuit operating voltage.

Another embodiment of a third submode (Mode 3″″′) further comprises amode in which the processing unit is operated at a substantially zerofrequency processing unit clock frequency (processing unit clockstopped) and at a processing unit core voltage just sufficient tomaintain processor unit state, and the logic circuit is operated at asubstantially zero logic circuit clock frequency and at a logic circuitoperating voltage that is just sufficient to maintain logic circuitoperating state.

One embodiment of a fourth mode (Mode 4) comprises a mode in which theprocessing unit is powered off by removing a processing unit clockfrequency (processing unit clock stopped) and a processing unit corevoltage.

An embodiment of a fourth submode (Mode 4′) further comprises a mode inwhich the processing unit is powered off by removing a processing unitclock frequency (processing unit clock stopped) and a processing unitcore voltage; and the logic circuit is powered off by removing the logiccircuit clock and by removing the logic circuit operating voltage or bysetting the logic circuit operating voltage below a level that willmaintain state, except that a real-time clock and circuit for waking thelogic circuit and the processing unit are maintained in operation.

Another embodiment of a fourth submode (Mode 4″) further comprises amode in which the processing unit is powered off by removing aprocessing unit clock frequency (processing unit clock stopped) and aprocessing unit core voltage; and the logic circuit is powered off byremoving the logic circuit clock and by removing the logic circuitoperating voltage or by setting the logic circuit operating voltagebelow a level that will maintain state, except that a circuit for wakingthe logic circuit and the processing unit are maintained in operation.

Some of the characteristics of these modes and submodes are listed inTable II. FIG. 15 provides an exemplary state engine state diagramgraphically illustrating the relationships amongst the modes andidentifying some of the transitions between states or modes foroperation of an embodiment of the inventive system and method. Note thatalthough the state engine may provide a path for directly or indirectlytransitioning between any two modes or submodes, in the interest ofkeeping the state diagram intelligible, the state diagram of FIG. 15does not show all of the possible state or mode transitions possible.

Having described several power or energy consuming states or modes (ortheir opposite, power or energy conserving states or modes) as well as asituation in which a hypothetical computer system may transition betweenthese modes, it will be appreciated that some procedure, mechanism, orpolicy is provided for the processor to self- or locally-control its ownoperating mode and hence its power consumption.

It is further noted that these operation modes may be utilized indifferent combinations and that any single system need not implement allof the operational modes. Therefore it will be appreciated that in theappurtenant claims, references to various modes, such as first mode,second mode, third mode, fourth mode, or the like, may refer tooperating modes or states in a general manner as otherwise defined inthe claims rather than to operating modes described in such terms in thespecification. For example, in the claims where two operating modes arerecited, such as first and second modes, such two modes may be any ofthe modes or states described, references, or suggested herein. TABLE IISelected Example CPU and Core Logic Clock and Voltage Ranges for VariousExemplary Computer System Operating Modes. CPU Mode Clock CPU Corevoltage Core Logic Clock Core Logic Voltage 1 ≈max ≈max ≈max ≈max 2 <max≦max ≈max ≈max 2′ <max <max ≈max ≈max 2′′ <max <max ≈max ≈max 2′′′ <max<max and sufficient to ≈max ≈max and maintain switching rate >0 (and CPUstate) 2′′′′ ≈min ≈min and sufficient to ≈max ≈max and >0 supportswitching rate (and CPU state) 3 <max ≦max and ≈min sufficient ≦maxbut >0 ≦max but >0 and ≧0 to maintain CPU state (typically = 0) 3′ ≈0≦max and ≈min sufficient ≈max ≈max to maintain CPU state 3′′ ≈0 <max and≈min sufficient ≈max ≈max to maintain CPU state 3′′′ ≈0 <max and ≈minsufficient <max ≈max, or sufficient to to maintain CPU state supportcore logic clock freq. 3′′′′ ≈0 <max and ≈min sufficient ≈0, except thatgenerally ≈max to maintain CPU state RTC remains active 3′′′′′ ≈0 <maxand ≈min sufficient ≈0, except that generally <max and ≈min sufficientto maintain CPU state RTC remains active to maintain logic state 4 =0 =0most core logic circuits most core logic circuits receive no operatingclock receive no operating voltage 4′ =0 =0 core logic circuits receivecore logic circuits receive no clock except for RTC no voltage exceptfor RTC and wake-up circuit and wake-up circuit 4′′ =0 =0 core logiccircuits receive core logic circuits receive no clock except for RTC novoltage except for RTC and wake-up circuit and wake-up circuit

Heretofore, control of the operating mode of a plurality of processorsor CPUs by a single supervisor or manager has not been known,particularly when the supervisor or manager is itself or includes aprocessor or CPU, and more particularly, it has not been known toprovide this type of multi-processor power management in a multi-serversystem. This level of control is referred to herein as global controlover a plurality of processors to distinguish from the afore describedsingle processor or CPU power management. It is noted that the inventivesystem and method also extend beyond any single-board computer systemshaving multiple processors configured therein. No such multi-CPUcomputers are known that provide power conservation features of the typedescribed herein, and it is noted that in exemplary embodiments of theinventive system and method that each of the plurality of processors arelocated within separate PC-board mounted module. Embodiments of theinventive system and method are provided for which both local-controland global-control are provided. Such global control over a plurality ofcomputers or appliances (each itself having either a single or multipleCPUs or processors) is not therefore limited to computers operating aservers.

Embodiments of the invention provide for detecting activity (orinactivity) in numerous ways, including but not limited to at leastthree different ways described herein. Detection may occur at the locallevel so that local control can be effected as well as optionaldetection at a global level. It is noted that in at least someembodiments, local detection of activity within each processor or CPUprovides sufficient information to globally control the powerconsumption of a system having a plurality of processors or CPUs.

In one embodiment, an OSI model having a physical layer is used foractivity or inactivity detection. In a second embodiment, a TCP/IP layeris used for this detection, and in a third embodiment the activity orinactivity detection occurs at the application layer. In a fourthembodiment, two or more of these activity detection and controltechniques are combined.

One technique for detecting activity or inactivity in the physical layeruses idle thread detection. In certain operating systems prevalent inthe late 1990's through 2001 provide a procedural “hook” through anoperating system functional call or other programing construct thatallows query of the operating system and generation of a response orreport back to the requestor indicating how much idleness is present inthe system, or more particularly how much idleness is present in theprocessor or CPU on which the operating system is executing. Thisoperating system query may for example be made using an API functioncall that returns a value. Some exemplary alternative techniques andprocedures for determining idleness in a system utilizes somewhatheuristic idleness detection algorithms, such an approach is describedin co-pending U.S. patent application Ser. No. 09/558,473 as well as inU.S. Pat. Nos. 5,396,635, 5,892,959 and 6,079,025 (each of which isherein incorporated by reference) by the inventor of the presentapplication as well as in the other applications related thereto.

With reference to FIGS. 16-23, several exemplary mode or state diagramsare illustrated. In these diagrams, a mode or state is represented by acircular node and a transition between two modes is represented by adirectional line or arrow, the arrowhead indicating the direction of themode transition. It is assumed for purpose of this discussion that thesystem may be in any one of three modes (Mode 1, Mode 2, or Mode 3) anda powered-off mode (Mode 4) (not shown). Some systems, such as certainTransmeta Crusoe™ CPUs operate so as to provide a maximum CPU corevoltage and maximum CPU clock frequency in a Mode 1 type operation and aplurality of levels (15 levels) in a Mode 2 type operation, theTransmeta Mode 2 operation consuming less power in fifteen of itsoperating levels than in the sixteenth operating level. Each of thesefifteen lower power consuming levels at which the CPU core voltage andCPU clock frequency are less than their nominal rated maximum areconsidered to be Mode 2 operating states as the processor operates in atmost one of the states at any given time and each separately qualifiesas a Mode 2 operation relative to the maximum performance Mode 1 stateand CPU suspend Mode 3 state. A mode state diagram for the TransmetaCrusoe LongRun™ CPU operation is illustrated in FIG. 18.

It is also noted that the Intel SpeedStep™ technology involves the sameor similar three modes of operation. The Intel SpeedStep provides for afully on mode running at maximum clock frequency and maximum CPU corevoltage, it also has a reduced state in which frequency and voltage arereduced relative to maximum, and a suspend state. During normaloperation such as for an AC-line powered notebook computer, the CPUclock frequency and CPU core voltage are at their rated maximum values.However, in at least one notebook computer made by IBM (IBM ThinkPadT21) a user may enable an optional power saving policy for batterypowered operation and for AC-line powered operation in which the CPUclock frequency and the CPU core voltage are reduced to save power andlengthen battery life. These power saving policies also control harddisk drive, display brightness, and the operating condition of otherinternal circuits and peripherals.

Each of FIGS. 16-23 shows a first mode (Mode 1), a second mode (Mode 2),and a third mode (Mode 3). A fourth mode (Mode 4) represents a processoror CPU that is powered down or in an Off state and is not shown. Variousmode transitions are supported by the inventive system and method.Conventionally, the transitions between and among the three modes werecontrolled locally (though such terminology was not used for suchconventional systems because there was no global control to contrastwith) because all or substantially all control was provided eitherwithin the CPU or by chips, logic, or other circuits associated with thesingle computer or PC-board on or in which the CPU was located. Inaspects of the present invention, global control is exercised over theoperating modes of a plurality of the processors or CPUs, and somedegree of local control is or may optionally be provided. The manner inwhich the transitions are controlled locally and globally are describedin greater detail elsewhere in this specification.

Recall that in single processor or single CPU systems, Mode 1 and Mode 2represent active work producing operating states, a non-zero frequencyprocessor clock signal causing the switching of transistor or othercircuits that permit instruction execution. Therefore, in singleprocessor systems, particularly in notebook computer systems operatingfrom finite energy sources (e.g. battery), the systems occupy most ofthe time they are “on” in a Mode 1 condition (or Mode 1-like condition)or in a Mode 2 (or Mode 2-like condition). Operation in a Mode 3condition does not provide any productive work so that if the user wereto perform any reasonable amount of work using the device containing thepower managed processor or CPU, there is little power savings that wouldbe achieved during useful work.

In FIGS. 16-23 the following notation is adopted. Each transitionindicating arrow is labeled with either an “L” to indicate localcontrol, a “G” to indicate global control, or an “LG” meaning that thetransition may be controlled by either or both local control or globalcontrol. In addition, transitions from Mode 1 to Mode 2 are labeled “A”and transitions from Mode 2 to mode 1 are labeled “A′★. In analogousmanner, other transitions are labeled as B, B′, C, and C′. This notationwill be useful in describing the differences between conventionalsystems and method and the present invention.

With respect to FIG. 16, there are shown locally controlled transitionsbetween Mode 1 and Mode 2 (A and A′) and between Mode 2 and Mode 3 (Band B′). For recent power management schemes, the A and A′ transitionswould normally be expected to occur with reasonable frequency during useof the notebook computer, and the B and B′ transitions with lowerfrequency, under the assumption that the user will typically either beusing the computer (A and A′ transitions) or power it off (Mode 4), sothat B and B′ transitions will be less frequent. It may also be expectedthat the B′ transition may be less frequent than the B transition, ascomputer makers may typically transition directly to Mode 1 from a Mode3 (C′ transition) when there is suddenly a need to wake up the CPU froma suspend type state. It is noted that for embodiments of the presentinvention, the B and B′ transitions may be frequent to very frequent,particularly when the 3rd mode is the Mode 3′ state in which only theCPU clock is halted and all or most other system clocks remainoperational. The Mode!3′ to Mode 2 (or Mode 1) and the Mode 2 (orMode 1) to Mode 3′ transition can occur very rapidly and because of thehigh CPU clock frequency and the number of switching circuits present inmodern CPUs can yield considerable power or energy savings. Embodimentsof the invention may also provide that a system operating in Mode 3′(CPU clock stopped or slowed significantly) may also further transitionto a Mode 3″ (CPU and other clocks stopped or slowed significantly)under specified conditions.

FIG. 17, illustrates an operating scenario under which the processor orCPU is maintained in an active state and only the A⇄A′ transitions occurunder local control. The B⇄B′ and C⇄C′ transitions are illustrated indashed lines.

FIG. 18, illustrates a similar operational scenario wherein theprocessor or CPU may transition to any one or sequentially through aplurality of Mode 2 states. This operational scenario is similar or thesame as the scenario under which the Transmeta Crusoe processor mayoperate.

The inventive architecture, system, device, and method may be operatedin a fundamentally different manner, using either only global control orusing a combination of local and global control, to alter the operatingmode of a plurality of processors or CPUs. Variations on this powermanagement scheme are now described relative to FIGS. 19-23.

In FIG. 19, the Mode 1 to Mode 2 A⇄A′ transitions are locallycontrolled. For example, in the Intel SpeedStep™ CPUs the A⇄A′transitions are controlled using control mechanisms provided by Intel ontheir CPU chips that permit a system designer to issue a command to theCPU to transition it from Mode 1 to Mode 2 under an identified conditionand from Mode 2 to Mode 1 under a second identified condition.Similarly, the Transmeta Crusoe CPUs implementing their LongRuntechnology would transition from Mode 1 to a selected one of a pluralityof Mode 2 states, and from that Mode 2 state (or a different Mode 2state) to Mode 1, under identified conditions. These conditions areknown in the art, available from Intel or Transmeta, or from Intel, AMD,or Transmeta computer manufacturer OEMs, and not described here ingreater detail

While the conventional systems and methods may permit the B⇄B′transitions and/or the C⇄C′ transitions under local or self-controlwithin a processor or CPU (or within circuitry associated with a CPU ona common mother board or other platform or enclosure), embodiments ofthe inventive system and method preclude such local or self-control.Rather, a manager or supervisor (see description of manager orsupervisor capabilities and implementations elsewhere in thisspecification) only may globally manage the B⇄B′ transitions and/or theC⇄C′ transitions under a global control scheme. Global control in thismanner is illustrated for example, in the state diagram of FIG. 20.

In yet another embodiment of the invention, depicted in the FIG. 20state diagram, Mode 2 operation is not supported and there are no A⇄A′transitions or B⇄B′ transitions. It is observed that operating only inMode 1 or Mode 3 would not represent a generally useful power managementscheme for a single processor or CPU system because Mode 1 operation isa full power active mode and Mode 3 is power conserving but inactivemode. Therefore, there is little power savings that would result whereCPU or processor loading is sufficient to keep the processor or CPU outof Mode 3. Significantly, systems or power management policies providingonly C⇄C′ transitions for single CPU systems (or for any processor orCPU systems) do not seem to exist in the computer industry.

On the other hand, this operating scheme is viable and presentssignificant power conservation features for multi-processor or multi-CPUarchitectures, particularly in the server environment where some orsignificant over-provisioning of server capacity is the norm and wherethe server suite may typically operate at from twenty to fifty percentof maximum capacity. As described in greater detail elsewhere in thisspecification, in the inventive Integrated Server System Unit (ISSU) aplurality of server modules, each having a processor, are integratedinto a single enclosure and coupled for communication by various in-bandand out-of-band bus and interconnection links. A manager or supervisoris provided (for example, in the form of a Management Module ordesignated Server Module operating as the manager or supervisor) thatcollects and/or analyzes CPU “activity” (where activity is definedbroadly as described elsewhere in this specification) and generatescontrol signals that maintain or alter the operating mode of individualServer Modules or identified groups of such Server Modules. While theprimary control is over the processor or CPU within these ServerModules, it is noted that other circuits or components, such as forexample, display, hard disk drive, and other circuits and/or peripheralsmay be similarly controlled by the same or different control signals.

Servers, server systems, or so called server farms generally designedand implemented with significant capacity over-provisioning. Reasons andrationale for such over provisioning is known in the art and thereforedescribed only briefly here. Providing a positive first visit Internetweb experience and maintaining a quality of service (QoS) is importantfor developing and maintaining clients, customers, or other visitors toa web site. Content must be served within a reasonable period of time,on a first visit and on subsequent visit, or visitors will not return.While the quality of service may be permitted to vary somewhat by timeof day and/or season, the reasonableness standard still applies, andnormally it is best to maintain a very high quality of service all thetime. Paramount in this goal would be to serve content such as webpages, streaming video, or cached content, without delay. Even duringtime periods (time of day, season, event driven) where web traffic andthe amount of content that need be served by a server is likely toincrease, sufficient server capacity must be in place. Over provisioningby at least 30% or so is typical, and frequently 100%-500% or moreover-provision or over-capacity may be provided.

This moderate to significant over-provisioning is accepted by the servercommunity as a necessary cost item, both in terms of the cost topurchase and maintain the equipment, the cost to power the equipment,the cost to cool or remove the heat generated by the equipment, and thenegative impact on equipment longevity as a result of continuousoperation.

Conventional server systems have not been power managed as there hasbeen a philosophy that if the equipment is there it should be operatedat maximum speed so as to serve content or respond to other requests asrapidly as possible. Conventional server units within a rack of serverunits have been to the inventor's best knowledge maintained in an alwayson always ready to serve mode. More recently, there has began to be someappreciation that power saving features provided in commercial personalcomputers might result in some power conservation benefits. At mostthese recent ideas have concentrated on the Mode 1 to/from Mode 2 (A⇄A′transitions) based on the Intel SpeedStep™, Transmeta Crusoe LongRun™,or other similar technologies. This local self-control by each processorprovides some energy conservation but does not provide the conservationof the inventive system and method.

One of the Transmeta Crusoe Model chips operates at 533 MHz and 1.6volts when in Mode 1 and at 300 MHz and 1.2 volts when at its slowestCPU clock frequency and lowest CPU core voltage in Mode 2. (Note thatthese operating parameters are nominal and subject to change by theirmanufacturer from time to time as products change, even within aparticular product model or family.) Recall that to a generalapproximation P∝K₁Cfv²+K₂, where P=power consumption, f is clockfrequency, v=CPU core voltage, C=capacitance, K₁ is some multiplicativeproportionality constant, and K₂ is some additive constant thatrepresents the small power consumed by a circuit when operating voltage(e.g. Vcc) is applied but the CPU or processor clock is turned off (e.g.0 MHz clock, or very slow clock). While these values may change fordifferent CPU designs and chip sets it will be clear that the savings intransitioning from a 1.6 volt/533 MHz operation to a 1.2 volt/300 MHzoperation is modest as compared to transitioning from a 1.6 volt/533 MHzoperation to a 1.2 volt/0 MHz operation. Operation with a CPU corevoltage that is equal to that of the CPU clock slowed Mode 2 or an evena lower CPU core voltage than that needed to maintain a 300 MHz clockswitching may be used during Mode 3 operation when only CPU register andmemory contents or status need be maintained.

It will therefore readily be appreciated in light of this descriptionthat operating a multi-server system where at least global control ofthe operating modes of a plurality of CPUs (and optionally other circuitelements of the servers) will yield significant power conservationbenefits. Furthermore, in some operational situations combining Mode 1to/from Mode 2 (A⇄A′ transitions) either locally controlled or globallycontrolled may add even further power conservation features.

FIG. 19 illustrates the state transition for an inventive embodiment inwhich A⇄A′ transitions are controlled locally, and B⇄B′ and C⇄C′transitions are under the control of a global manager. FIG. 20illustrates the state transition for an alternative inventive embodimentin which the processor or CPU only operates in either Mode 1 or Mode 3and not in Mode 2 so that A⇄A′ and B⇄B′ transitions are prevented fromoccurring (such as by disabling a feature provided with a chip,de-configuring power conservation features, or providing the managerwith the ability to otherwise prevent such transitions), and C⇄C′transitions are under the control of the global manager.

FIG. 21 illustrates the state transition for yet another alternativeinventive embodiment in which the processor or CPU only operates in anyof Mode 1, Mode 2, or Mode 3 and while the A⇄A′ transitions occur underlocal control, the B⇄B transitions are prevented from occurring, andC⇄C′ transitions are under the control of the global manager. In thisembodiment, therefore, the transition to Mode 3 therefore only occursdirectly from Mode 1 and never from Mode 2. In yet a further embodiment,illustrated in FIG. 22, the A⇄A′ transitions occur under local controland the B⇄B′ transitions occur under global control, and where C⇄C′transitions do not occur. FIG. 23 illustrates the mode transitions in afurther embodiment, where each of the A⇄A′, B⇄B′, and C⇄C′ transitionsmay occur according to predetermined power management policies and whereeach separate possible transition may be under either local and/orglobal control according to the predetermined policy or power managementprocedure or algorithm. The policy, procedure, or algorithm may alsodisable certain states of transitions statically or dynamically, and maycause certain of the server modules or other CPU or processor baseddevices into a powered off (Mode 4) and back to any of the powered onmodes.

FIG. 24 illustrates that for a system having a plurality of processor orCPU based devices, the CPU or processor within any particular device(such as server modules) may be in different states at different timesunder the direction of an device-local control, a system supervisoryglobal control, or a combination of the two. The shaded mode circlesindicate the current mode and the mode transitions, though not shown,may be any of those already described relative to the other inventiveembodiments.

In light of the above description, it will be appreciated that theinventive system and method extends earlier power management structures,architectures, and methods by the same inventor Henry T. Fung (such asare described in U.S. Pat. Nos. 6,115,823; 6,079,025; 5,987,614;5,961,617; 5,892,959; 5,799,198; 5,758,175; 5,710,929; and 5,396,635,herein incorporated by reference) to multi-server or multi-nodearchitectures.

These existing power management patents include innovative systems,architectures, and methods for saving or conserving energy or powerwithin a single system by using one or more of several power managementschemes, including, but not limited to the following schemes: (1)Detection of the idle activities by monitoring I/O activities orexecution of a predefined code thread. (2) Reduction of powerconsumption by lowering (or stopping) various clock frequencies orremoval of power (operating voltage) to different components within thesystem. (3) While in a power saving mode, continuing to monitor theoccurrence or non-occurrence of a second predefined event or activityand entering a deeper power saving mode in response to the secondpredefined event or activity detection. Note that although certainevents, activities, and/or indicators are referred to predetermined,such events, activities, or indicators may be dynamically determinedduring operation as well as determined in advance.

The present Multi-Server Power Management scheme extends these earliertechniques, augments them, and introduces entirely new features andcapabilities. Five particular innovations are set forth below, however,it will be apparent that the invention described herein is not limitedonly to this set of features and capabilities.

First, power management of the network devices including the servermodules can occur at different OSI levels and be extended beyond thephysical layer. In particular, the detection of server activity whethermeasured by idle activities or other means may occur at the physicallayer but is advantageously extended beyond the physical layer to thenetwork layer (for example, to the TCP/IP layer) and to the applicationlayer. For example, at the physical layer, the number of CPU idlethreads within a fixed time period may be detected or measured, or, sometype of statistical evaluation of CPU idleness may be determined. As onenumerical example, if the CPU is idle 80% of the time while in aparticular operating mode such as Mode 1, it is clear that this muchprocessing performance is not required and the CPU performance maytherefore be adjusted downward to save power. If we assume in a simplecase that a Mode 2 operation reduces the CPU clock speed by a factor of¼ over the Mode 1 clock speed, then the CPU will only be able to process¼ of the instructions in the same period of time, however, this issufficient given the 20% loading (80% idleness) the CPU is experiencing.Therefore, based on this idleness detection, significant power savingsare realized. Alternatively or in addition, if for example, under thesame scenario there is a group of ten network server devices that arebeing managed as a single logical group or image, eight of them may beput into an inactive but powered on Mode 3, and the other two networkserver devices placed in a Mode 1 operating state running at a 100%performance level.

Power management may also or alternatively occur based on detection atthe TCP/IP layer (or equivalent layer where a protocol other than TCP/IPis implemented). Under this detection and control model, CPU performanceis monitored relative to the handling of TCP/IP packets. CPU performancelevel is lowered, such as by reducing CPU clock frequency (desirablyaccompanied by a reduction of CPU core voltage) until packets startdropping, and then increasing performance so that packets are notdropped and to provide an operating margin. The initial reduction andsubsequent increase in CPU or server performance may be accomplished byaltering the operating mode of individual servers or by adjusting theaggregate characteristics of a group of servers to provide the aggregateperformance required. It is noted that where communications channelbandwidth limits the performance of a server, there may be advantage toreducing the performance level of the server to just satisfy thebandwidth limitation and thereby conserve power in the server.

At the application layer, the activity monitoring or detection may forexample involve measuring the number of times a specific port address isor has been requested within a fixed time period. This determination ormeasurement may be accomplished, for example, by using a SNMP agent. Inresponse to this measurement, an appropriate number of servers eachoperating at an appropriate performance level (Mode 1 or Mode 2) areprovided to meet the performance requirement for each application. Therest of the servers are placed in a highly power saving state (Mode 3such as Mode 3′ [e.g. CPU clock halted] or Mode 3″ [e.g. CPU and otherlogic clock stopped], or Mode 4). The policies for selecting the numberof active servers and their operating mode are described elsewhere inthis specification. Recall that different application types may usedifferent rules or policies to determine the server CPU performance andpower conservation requirements.

Second, power management is extended beyond a single processor of CPUand in particular is extended beyond a single server (independent of thenumber of processors it may contain) to multiple servers across anentire network. It will be appreciated that this multi-server powermanagement capability may be provided either with discrete servers orwith the particular embodiment of the Integrated Server System Unit(ISSU) or Integrated System Server architecture generally.

Third, activity information created by any one server (or server modulein the ISS scheme) is accessible to a designated supervisor via standardnetworking protocol. This supervisor is frequently referred to as themaster, the capabilities of the master residing for example in an ISSManagement Module or an ISS Server Module, though the particularlocation or processor responsible for accessing and utilizing theactivity information for the servers is not critical to the powermanagement. In preferred embodiments of the invention, the supervisor ormaster capabilities reside in one or more management modules, and in analternative embodiment, the supervisor or master capabilities reside ina designated or selected one of the server modules.

Fourth, servers can be reconfigured to run a specific application (e.g.web, streaming media and email) based on a certain load distributionrequirement or requirements existent at the time upon receiving commandsfrom a designated supervisor or master. Advantageously, this featurewill provide or support operation at three or more power consumptionlevels, including a first full power mode (full CPU core voltage andnormal maximum CPU clock frequency), a second mode consuming less powerthan the first mode in which either the CPU core voltage or the CPUclock frequency or both are reduced from the first mode, and a thirdmode in which the CPU is substantially inactive and consumes less poweror energy than the second mode. In one embodiment, this third modeprovides a CPU core voltage to maintain state and either stops the clockor maintains the clock at a very low frequency (for example, 1 Hz to afew hundred Hz) so that the CPU is effectively inactive.

Fifth, allowing any number (including none, one, many, or all) ofservers across the entire network to go in and out of a 3rd powerconsumption mode directly from a first mode (Mode 1) without goingthrough another intermediate power saving mode upon receiving commandsfrom a designated master. This third power consumption mode (Mode 3) mayfor example include a mode where the processor or CPU is powered at somelevel but substantially inactive from the standpoint of executingcommands or serving content, and memory associated with the CPU isrefreshed. This third mode may be further broken down into a mode inwhich only the CPU clock is stopped (Mode 3′) such as may occur when aHalt instruction is executed, and into a deeper power savings mode inwhich the CPU clock is stopped and other clocks are also stopped (Mode3″). It is noted that in a typical implementation, the real-time clock(RTC) will generally run al the time so that certain system timingevents and alarms can be maintained. The third power saving mode mayalso or alternatively be a powered down mode (Mode 4), however, suchoperation is somewhat undesirable unless it is anticipated that thepowered down (Mode 4) server module will not be needed for someappreciable period of time as a delay is associated with bringing theCPU and the server module within which the CPU is located back on line.The Mode 4 operation may therefore only be used when the Mode 4operation is expected to continue for several seconds, minutes, hours,or longer periods of time. It will be appreciated that in the thirdpower saving mode, the CPU clock (and or other clocks in the system) maybe either off entirely or running at a very low rate (such as forexample 1 Hz, 10 Hz, 100 Hz, 1 KHz, or some other value that is small incomparison to the nominal frequency (for example, typically in the 100MHz to 2 GHz range) of the processors used for such servers. It will beappreciated in light of the description provided here, that theinvention provides for direct transition between a full or substantiallyfull power mode and an inactive or substantially inactive mode.Although, this power mode transition would be much less useful forbattery-powered portable applications for notebook computers or PDAsbecause of the desirability of maintaining some activity such as whentyping into a word processor, this transition scheme extremely useful ina multi-server environment, where each of a plurality of servers canserve the same content and it is desired to reduce the number of activeservers while maintaining sufficient ability to satisfy quality ofservice requirements or otherwise maintain operation with a subset ofthe total set of servers.

These five innovations (as well as others) may of course be combined invarious ways to provide even greater synergism. For example, the firstdescribed innovation extending the detection of idle activities beyondthe physical layer to the network layer and/or to the application layer,may readily be combined with the fourth described innovation wherein theservers can be reconfigured to run a specific application based on acertain load distribution requirement or requirements existent at thetime upon receiving commands from a designated supervisor or master.

This combination may also be extended according to the second describedinnovation to include multiple servers across an entire network,independent of whether the servers are discrete or integrated ISSU-basedserver modules. This latter combination may be further enhanced by alsoimplementing the third described innovation to provide that activityinformation created by any one server (or server module in the ISSscheme) is accessible to a designated supervisor or master via standardnetworking protocol.

In yet another embodiment, the filth described innovation that providesfor any number of servers is a system having a plurality of servers totransition directly from a full performance 1st mode to an inactive 3rdmode. This scheme generally representing a non-useful power managementscheme when applied to any single computer or server, but providingconsiderable benefit when the plurality of servers are managed incombination to provide a desired level of performance and powerconsumption savings.

Table III describes the behaviors of selected component inside anexemplary computer system, such as a computer system configured as aserver module, at the different power management modes (Modes 1, 2, 3,and 4) according to one embodiment of the invention. This embodimentimplements somewhat different power management policies than theembodiment described relative to Table II and also addresses the mannerin which certain other peripheral devices or other components may bepower managed. The mode descriptions are therefore generically similarbut the detail or submode descriptions differ somewhat, but suchdifferences are semantic and each of the modes and submodes described inany of the embodiments are within the scope of the inventive system,apparatus, computer program, and method.

In this embodiment's first mode (Mode 1) the processor or CPUfunctionally able to execute instructions for operating system andapplication programs; CPU activities are monitored, and the internal CPUclock frequency and CPU core voltage may be lowered if activity level ofthe CPU falls below some threshold (predefined or dynamically determinedthreshold). The voltage regulator is set to deliver the maximum (orspecified nominal) CPU core voltage, the clock generator, RAM, hard diskdrive, core logic, NIC, BIOS, and Real-Time Clock (RTC) are ON. Thevideo may independently be controlled to be on or off and may even beabsent from the system as video signals frequently are not needed forserver systems, except in some cases of set-up or service. Amicrocontroller (μC) is operative and remains in continuouscommunications with the Management Module (or with an different ServerModule designated or selected to operate as a manager or supervisor.

In Mode 2, the CPU still executes operating system and applicationprogram instructions, CPU activity is still monitored, and if theactivity level rises above some predetermined or dynamically determinedthreshold (or according to some other rule or policy) the CPU entersMode 1 operation, but the CPU enters Mode 3 in response to receipt ofMode 3 entry commands received from a manager of supervisor. These Mode3 entry commands may generally be received from an external master viastandard in-band network protocols. Recall that in Mode 2 the voltageregulator that supplies CPU core voltage is set to less than maximumcore voltage. As in Mode 1, the clock generator is on but will (inpreferred embodiments) deliver a lower frequency clock signal, and RAM,hard disk drive, core logic, NIC, BIOS, and Real-Time Clock (RTC) areON. The Video may independently be controlled as in Mode 1. Amicrocontroller (μC) is operative in Mode 2 to activate a suspend signalof the core logic power management unit or PMU (Out of Band) afterreceiving commands from the management module (or server module actingas a designated master or manager) and causes the particular serverand/or multiple servers within the system to enter the 3rd mode. TABLEIII Exemplary behaviors of selected components inside a computer system(e.g. server module) at the different power management modes accordingto one particular embodiment of the invention. Other embodiments supportalternative or additional modes and transitions between modes asdescribed for example in Table II. 1^(st) Mode 2^(nd) Mode 3^(rd) ModeCPU 1) Execute applications 1) Execute applications 1) CPU is in verylow power 2) Monitor CPU activities 2) Monitor CPU activities state 3)Lower internal CPU 3) Go to the 1^(st) mode if 2) Return to 2^(nd) modeor clock frequency and activity level rises above a 3) Return to 1^(st)mode voltage if activity level pre-defined threshold falls below apre-defined 4) Go to the 3^(rd) mode after threshold (go to 2^(nd) mode)receiving commands from an external master via standard network protocol(In Band communication) Voltage CPU core voltage is set to CPU corevoltage is set to CPU core voltage is set to be Regulator maximum lessthan maximum equal to or less than core voltage in 2nd mode setting.Clock ON ON Stop most (or all) clocks. Generator For example, may stoponly CPU clock, or may stop CPU and other clocks. (Usually RTC is notstopped.) RAM ON ON Suspended (refresh only) Hard Disk ON ON Suspendedafter receiving commands from the CPU Core Logic ON ON Suspended afterreceiving commands from the CPU or signal from uC NIC ON ON Suspendedafter receiving commands from the CPU or turning off NIC Clk. Sendresume signal to core logic after a predefined packet is received (e.g.Wake-On- LAN) Video ON/OFF ON/OFF Suspended after receiving commandsfrom the CPU or turning off Video Clk BIOS ON ON Suspended RTC ON ONSend resume signal to the core logic after alarm expire microcontrollerContinuous Activate the suspend signal Send resume signal to core (UC)communications with the of the core logic PMU (Out logic after receivingmanagement module. of Band) after receiving commands from the commandsfrom the management module management module and causes the entiresystem to enter the 3^(rd) mode

In Mode 3, the CPU is placed in a very low power consumption state andcan return to Mode 1 or to Mode 2 upon the occurrence of somepredetermined condition such as are described elsewhere in thisspecification. The voltage regulator that provides CPU core voltage isset to a voltage equal to or less than the core voltage in Mode 2 tothereby save power over that consumed in either of Modes 1 or 2. Theclock generator is also stopped so that power consumed switching devicesis substantially eliminated. (It is noted that in an alternativeembodiment, the clocks in Mode 3 may be operated as a very slow rate,for example a few Hz to a few hundred Hz, or some other low clockfrequency relative to the normal clock rate of the CPU.) RAM issuspended (that is the memory contents are refreshed only), the HardDisk drive or drives are suspended after receiving commands from the CPU(or other commands to spin down and go into a suspend state). The corelogic is also placed into a low power consuming suspend state afterreceiving a command from the CPU or signal from the micro-controller.Mode 3 operation also provides for suspension of the networkinterconnect card or circuit (NIC) after receiving commands from the CPUor turning off the NIC clock. (Note that a resume signal may begenerated and sent to the core logic if a predefined packet is received,such as for example, a Wake-On-LAN signal.) The BIOS is suspended, andthe RTC may send a resume signal to the core logic after a RTC alarmexpires. The microcontroller continues to monitor communications withthe management module or other designated master so that it may send aresume signal to the core logic after receiving commands directing thistype of action from the management module or other designated managementmaster. If the Video was on prior to entering Mode 3, the Video issuspended after receiving commands from the CPU or the Video Clocksignal is stopped or turned off, and if it was off it remains off.

While much of the description herein has focused attention onperformance and power management of the processor, CPU, core logic, andother logic circuits within a computing device or system, or otherinformation instrument or appliance having such processor and/or logic,it should be understood that the dynamic power management and dynamicworkload management is not only limited to such systems or components.More particularly, the inventive dynamic power management system,method, architecture, procedures, and computer programs may also beapplied to a diverse set of electrical and electronic componentsincluding components commonly referred to as computer peripherals.Application of the principles described herein therefore have thepotential of reducing power consumption and prolonging component life tosuch devices and systems as video monitors, hard disk drives or otherstorage systems or devices, printers, scanners, cameras, other networkdevices and circuits, industrial tools and systems, and a myriad ofother systems and devices.

Hard disk drive storage systems benefit from the inventive system andmethod as well as other inventive features as described in co-pendingU.S. Provisional Application Ser. No. 60/236,062 entitled System,Apparatus, and Method for Power Conserving and Disc-Drive LifeProlonging RAID Configuration filed 27 Sep. 2000; incorporated herein byreference, a system and method for operating and controlling a pluralityof rotatable magnetic hard disc drives operating in a Redundant Array ofIndependent Discs (RAID) was described.

Heretofore, magnetic hard disc drives had remained the primary computerdata storage and retrieval medium for may reasons, including: low costper megabyte of storage; very fast read and write access; multipleoperating system and disc drive device support; the ability to organizeinto arrays to provide either greater capacity, data redundancy, orboth; as well as numerous other advantages as are known in the art. Forthese and other reasons, disc (or disk) drives, particularly rotatablemagnetic hard disc drives have found application in a great variety ofdata and other information storage and retrieval applications.

Hard disc drive technology continues to evolve to provide higherrecording densities, greater disc drive storage capacities, higherspindle speeds, reduced seek time, faster burst and/or sustained datatransmission rates. Many disc drives are specialized for a particularapplication either in terms of storage capacity (typically 1 to 30gigabyte or more, physical size (e.g. 1.8-inch, 2.5-inch, 3.5-inch, or5.25-inch form factor), interface compatibility (e.g. ATA, IDE, SCSI,Fire Wire, to name a few), intended application (e.g. portable notebookcomputer, home or office desk top computer, commercial serverenvironment, instrumentation, as well as many other standard andspecialized applications).

Where the integrity of the data is of particular concern, such as incommercial server and/or database environments, some form of on-linedata redundancy is typically provided. For example, one or moreRedundant Array of Independent Disc (RAID) may be provided to providedesired storage capacity and data redundancy. The RAID was proposed inthe paper “A Case for Redundant Arrays of Inexpensive Discs (RAID)” byD. A. Patterson, G. Gibson, and R. H. Katz, Report No. UCB/CSD 87/391,University of California, Berkeley, Calif. 1987, incorporated herein byreference. Their basic idea for RAID was to combine multiple small,inexpensive discs into an array that outperforms a Single LargeExpensive Drive (SLED). This array of discs would be arranged in such away so they would appear to a computer as a single logical drive eventhough comprised of a plurality of physical drives. They calculated thatthe Mean Time Between Failure of this array would be equal to that of asingle drive divided by the number of drives in the array. Therefore,they defined 5 different array architectures, each providing disc faulttolerance and each having different characteristics in order to achievemaximum performance in different environments. An additionalnon-redundant architecture (RAID Level 0) was also defined. Various RAIDconfigurations or levels are presently known in the art and variationsof these standard RAID levels continue to evolve over time.

Of particular interest here is the so called RAID Level 1 (RAID-1) whichis also referred to as “mirroring” for reasons that will shortly becomeclear. The so called RAID 10 or RAID 0+1 is also of some interest as itinvolves mirroring in addition to data striping. A RAID-1 discconfiguration creates an exact duplicate to a second (or mirror) discany time data is written to a first (or primary) disc. Conventionally,this duplicate is created automatically and transparently to the system,application, and user. The user may not even be aware that the duplicatemirrored disc exists. Ideally, the mirrored disc is an exact duplicateof the data disc, though it is possible and acceptable for there to besome variation in the location and distribution of data between theprimary and mirrored discs. While mirroring desirably occurs on a secondhard disc drive so that the failure of the primary drive leaves themirrored drive in operation, in some instances, the mirror drive may bea second disc platter within a single hard disc drive housing. Suchsingle disc RAID-1 configurations are more susceptible to failure, andprovide the desired redundancy only when the medium for the primary discplatter fails but does not provide redundancy when for example, the discdrive spindle motor fails.

The interface to the primary and mirror RAID-1 drives can be through asingle controller which produces the performance of a single drive forreads and writes. Alternatively, two controllers (e.g. duplexing) may beprovided so as to reduce the single point of failure risk. Duplexing canimprove I/O data rate by allowing a zig-zag read or by writing to bothdrives simultaneously. When mirroring with a single controller, data iswritten first to the primary data drive and then to the mirrored drive.This generally slows down write operations.

Mirrored disc configurations are frequently used where high faulttolerance is required, such as in most fault-tolerant transactionprocessing systems. They attempt to improve the reliability of the discstorage device rather than improve data transfer rates. TheMean-Time-Between-Failure (MTBF) of a mirrored disc storage subsystemgreatly exceeds the expected life of a system with a single set of discdrives utilizing conventional non-mirrored configurations. For RAID-1mirrored system configurations significantly increase the probabilitythat data can be recovered in a drive fails. The biggest disadvantage isthat only half of the total disc storage capacity is available forstorage as each set of data is duplication. RAID-1 mirrored storagecapacity can only be expanded in pairs of drives. Of the various RAIDlevels, RAID Level 1 provides the highest data availability since twocomplete copies of all information are maintained. In addition, forconventional implementations read performance may be enhanced if thearray controller allows simultaneous reads from both members (primaryand mirror) of a mirrored pair. During writes in conventionalimplementations, there will be a minor performance penalty when comparedto writing to a single disc. Higher availability will typically beachieved if both discs in a mirror pair are on separate I/O busses, butthis is not required.

Data striping is the foundation of certain RAID levels, including RAID0+1. Disc drives in a RAID group are partitioned into stripes, which maybe as small as one sector or as large as several megabytes. The stripesare interleaved so that disc space is composed of alternate stripes ofeach drive. Data is written across the stripes instead of onto a singledrive. The sizes of the stripes vary depending upon the application. I/Ointensive applications benefit from large stripe sizes and dataintensive benefit from small stripe sizes. Data striping desirablyserves to balance the I/O load across all the disc drives in an array.With multi-user operating systems like Windows NT, Unix, and Netware,that support overlapped disc I/O across multiple drives, data stripingkeeps all the drives in the array busy and provides for efficient use ofstorage resources. In non-striped arrays the I/O load may not bebalanced. Some of the drives may contain a lot of frequently used fileswhile other drives may lay idle. Striping, when implemented, maytypically provide higher performance because all drives are involved asmuch as possible.

As described above, RAID Level 1 provides disc mirroring where data iswritten to a primary disc and a secondary (mirror) disc, and identicaldata is stored on both discs. RAID level 1 does not provide for datastriping. RAID Level 0+1 (RAID 10) is a combination of RAID level 0(data striping without mirroring) and RAID level 1 (mirroring withoutstriping) and provides for striping data across two pairs of mirroreddisc drives. This configuration may provide for high performance writeoperations as there is no parity overhead. Again, there is a somewhathigher cost per megabyte of storage because four physical disc drivesare required to achieve the capacity of two drives.

RAID Level 1 and. RAID Level 0+1 are generally recognized as goodchoices in data storage environments where performance and dataprotection are more important than cost. The cost penalty arising fromthe need to have a duplicate set of disc drives.

Unfortunately, the requirement to provide twice the number of physicaldisc drives as warranted by the actual storage requirement comes at aneven greater penalty than the cost of the disc drives themselves and theadditional penalty of having to perform two write operations, one to theprimary drive and the same data to the mirror drive. These penaltiesinclude the two-times increase in power consumption associated withoperating a duplicate (or set of duplicated) mirrored disc drives, theheat dissipated by the added drives within the enclosure which will tendto age other electronics components at a faster rate, the added noiseassociated with the additional disc drives, and the shortened effectivelife span of the drives. These penalties are experienced in spite of thefact that the primary disc drive may not fail so as to require orotherwise benefit from the mirrored disc drive, and further in spite ofthe probability that the mirrored disc drive may fail before the primarydisc drive. For identical discs, having the identical operating history,it is clear that each has substantially the same probability of failureand that either the primary or secondary drive may fail first.Therefore, there has remained a need for a system and method that reducepower consumption in a primary-mirror multi-disc drive system, as wellas a need for a system and method that extend the life of either or bothof the primary and mirror disc drives. There remains a further need fora system and method that reduces heat and noise in the computingenvironment.

FIG. 25 is a diagrammatic illustration of an exemplary computer systemwith which the inventive structure and method may used. Thisconfiguration represents what is commonly referred to as hardware RAIDcontrol and utilizes a separate hardware RAID controller with it's ownprocessor. A software RAID control configuration is also supported andillustrated in FIG. 26. A host computer 101 having a processor (such asa CPU) 102 coupled to a memory 103 generates I/O requests to a massstorage system such as a RAID storage subsystem 104. The I/O requestsmay be communicated over one or more buss 109. In the present invention,the RAID storage subsystem may be configured to include mirroring ofdata between two sets of storage devices. The storage devices maytypically include hard disc drives, but is not limited to such hard discdrive devices. The I/O requests or other commands are passed to thestorage subsystem 104, frequently through a RAID controller 108, butsuch storage subsystem control may alternatively be controlled by thehost computer. Usually, the presence of a separate storage subsystemcontroller (e.g. RAID controller) will depend on the I/O requirements ofthe system and the tolerable level of host loading. In server typecomputer systems one or more separate storage controllers are generallyused. In demanding server applications, multiple RAID controllers arethe standard. Host computers, storage device controllers including RAIDcontrollers, and storage subsystems including RAID 1 and RAID 0+1subsystems are known, and except for the inventive modifications andimprovements are not described in detail.

Whether provided by the host computer (software RAID) or by a separatehardware storage subsystem controller (hardware RAID), or some hybrid ofthe two, the inventive structure and method provide procedures 110 foroperating the storage devices, such as the RAID hard disc drives so asto reduce power or energy consumption and to increase effective discdrive life. These procedures may usually be stored in memory 112 (eitherhost memory or controller memory depending upon the implementation)during execution by the processor (either the host processor or thecontroller processor). Data 114 may also be stored in memory 112.Desirably, the storage device control procedures are stored as softwareand/or firmware in a non-volatile memory. The storage controller andhost computer may be any of the types available that support RAIDdevices and that may be customized or programmed to support theinventive procedures for activating and deactivating the disc drives asdescribed hereinafter.

In the software RAID implementation, the inventive procedures are storedin memory 103 of the host and executed by the host processor 102. Whileperformance may not be as high in extreme I/O environments, it may bemore than adequate to meet demand in any particular environment, and ismuch more cost effective as the additional expense of a separate RAIDcontroller is eliminated.

In many server environments, particularly in Internet serverenvironments where extremely large volumes of identical data may be sentto thousands or millions of browsers, most accesses to the disc storagesystem are read operations. For example, in a server serving currentinformation concerning athletes and competition in the Year 2000 SummerOlympics being held in Australia, the ratio of data write operations toread data operations on any particular server must be many thousands tomany millions to one. Even for an event where new results may appearhourly, write cycles occur relatively infrequently as compared to readcycles. For old information, the ratio of read operations to writeoperations may be much higher.

In the RAID 1 (and RAID 10) configurations, only one drive (primary ormirror) or one set of drives (primary set or mirror set) need to beavailable or powered ON at a time to support such read operations. Theidentical nature of the data stored on the primary and mirror drivesonly changes in response to a write operation and is unchanged by themore frequent read operations. One of the drives can be either in astandby mode (such as a mode where power is provided by the spindlemotor is not operating) or with operating power (e.g. operating voltageand/or current) completely removed. Various shades of power conservationmay be applied between completely on and completely off, particularly ifthe internal circuitry of the disc drive and control electronics and anyon-board buffer memory or the like are designed with staged powerconservation features. It is noted that since the primary and secondarydisc drives store identical data and are completely interchangeable froma functional (and likely from a physical standpoint) there is littleneed to identify that it is the primary or the secondary drive that ispowered off or placed into a standby mode, reduced power consumptionmode, power conservation mode, or simply powered off. More accurately,we may refer to the drives as the active drive (or active drive set) andthe inactive drive (or inactive drive set).

FIG. 27 and FIG. 28 illustrate the manner in which data is distributedamong disk drives (or disk drive sets) in RAID Level 1 and RAID Level0+1 respectively. In general the number of disks and the capacity ofdiscs may be selected to match the storage requirements.

Where one of the primary or secondary disc drives or disc drive sets arepowered off, the power consumption of the mass storage sub-system can bereduced by 50 percent, and where some power is provided to the discdrive or to the control electronics responsible for waking up the discdrive when needed, a proportionate decrease in power consumption will berealized. Frequently, merely removing operating power from the spindlemotor will have a significant effect on reducing power consumption, heatgeneration, and wear.

During a write cycle, which occurs very infrequently relative to readcycles, the CPU, operating system or other control means or process willrestore power to the inactive drive first before it starts writing tothe active one. Alternatively, some small delay may be tolerable betweena write to the active drive (or active drive set) and a write to theformerly inactive drive (or inactive drive set), but this is notpreferred as it leaves the system somewhat vulnerable to non-recoverablefailure and raises issues of data coherency and synchronization betweenthe active (e.g. primary) and inactive (e.g. mirror) data sets.Restoring the power to the inactive drive first will allow the inactivedrive enough time to come up to speed so there will be no dead time ordelay between writing of the two drives. Where feasible, writeoperations to the active drive may desirably be grouped so as tominimize to the extent possible the number of power-up and power-downoperations as these operations may accentuate drive wear in some discdrive systems.

While some disc drive structures may suffer somewhat greater wear whenthey are repeatedly power on and powered off, there are also disc drivesthat have their useful life drastically reduced if they have to stayactive continuously. More rapid wear from being powered ON and poweredOFF may in some instance result from wear that the disc surfaceexperiences from disc-to-transducer head contact during landing, and/orfrom spindle motor wear experienced as the spindle speed decreases andlubrication of the bearing contact surfaces diminishes. Rapid wear onother hard drive structures as a result of prolonged or continuous ONtime. Whatever the reason for this more rapid wear or increase in discdrive failure, the reduction in disc drive lifetime (decrease in MTBF)has been particularly pronounced in the 2.5-inch form factor hard discdrives typically found in notebook computers and other portablecomputing devices. It is also likely to be a problem in 1.8-inch formfactor drives however these size drives are not widely used.

Without benefit of theory, it would appear that some of the lifetimereduction of the 2.5-inch form factor drives is related to the lighterweight structures associated with high-density large disc storagecapacity in a small physical package. Design tolerances are tighter,components, including components that generate heat are closer together,and the likelihood of a failure is increased. While such 2.5-inch formfactor drives have heretofore been infrequently used in database andserver applications, the inherently lower poser consumption of such harddisc drives designed for potable computing makes them ideal candidatesfor smaller, higher capacity, lower power consuming server applications.

As an added benefit, the distinction between the primary drive (or driveset) and the secondary drive (or drive set) need not be fixed. Since theRAID 1 configuration requires only one drive (or one set of drives) tobe active for any given read operation, the other drive (alternatedrive) can essentially be shut off completely or put into power savingmode. At a predetermined time interval or according to some otherpredetermined rules (such as the number of write operations), the activeand inactive drive can exchange role, that is, the formerly active drivebecomes the current mirrored drive and the formerly mirrored drivebecomes the currently active drive. This technique will allow any onedrive to be active for a maximum 50 percent of the time. Consequently,the expected life time for both drives (or drive sets) will be extended.

While the description provided heretofore has focused primarily on RAID1 configurations, the principles also apply to RAID 10 (also referred toas RAID 0+1) as this RAID configuration provides data striping as wellas mirroring of the striped data onto the mirrored drive. For a RAID 10configuration, the data would be written in stripped fashion to theprimary and mirrored drives or drive sets in normal manner, but readoperations would only require participation of the currently active RAID10 drive or drive sets.

Furthermore, where even greater redundancy is desired, any plurality ofmirrored or mirrored and striped disc drives may be utilized. Forexample, if additional data redundancy in a mirroring type configurationwas desired, then mirror copies may be provided on two (or more) mirrordrives in addition to the data stored on the primary drive. Rules andprocedures for powering down or placing into an energy or powerconserving mode may be implemented so that one or both mirror drives arebrought back to an active state to accept a write operation update. Eachof the three (or more) drives may then be selected as the current activedrive according to some predetermined rules or procedures. In the threedisc drive example, each disc drive would at most be active for about 33percent of the time, significantly increasing the lifetime of all thedisc drives.

Finally, although the description has focused on RAID configurations, itwill be understood by workers in the art that the same principle may beapplied to other than RAID storage configurations where data redundancyis maintained on separate physical storage media. Furthermore, while thedescription has also focused on magnetic hard disc drives, the inventionis not so limited, but may be applied to any storage system whereredundant data is maintained on multiple storage devices. For example,the inventive system, apparatus, and method are applicable to rotatablemagnetic hard disc drives, optical disc drives, CDRAM, DVDRAM drives,solid state memory devices, or any other storage device or media wherethere is a desire to maintain some redundancy while reducing powerconsumption, heat generation, component aging or wear, noise, or othernegative effects.

One particular embodiment of the inventive method is now described.During an initiation procedure (such as may occur when the computersystem is booted or reset), and after the two (or more) alternative discdrive sets are synchronized such that they store the identical data (inconventional RAID 1 or RAID 10 manner), one of the drives (now theactive drive) is maintained in operating mode so as to service any readI/O requests, while the other drive (now the inactive drive) is placedin a power conserving mode. Power conserving mode as used here refers toany mode or state between a completely active powered on state where thedrive is ready to respond to a read or write I/O request and a poweredoff state or more. For example a power conserving mode includes variousstandby modes where the spindle motor has stopped but the dive is stillresponsive to control signals. Various industry power managementstandards are applicable to controlling hard disk drives and may beadapted to control the energy consumption state of such disc drives orother mass storage devices.

The active drive responds to any read I/O request. In the event that awrite I/O operation or some other operation necessitating a change inthe data or control information written to the active drive occurs, thesystem hardware and/or software/firmware activates the formerly inactivedrive or drive set to perform the write operation and maintain the twodrives or drive sets as identical copies. Once the write operation iscompleted, one of the drives is deactivated or otherwise placed into anenergy conserving mode or state. In one embodiment, the recentlyactivated drive is deactivated after the write operation and again theactive drive services read I/O requests. In an alternate embodiment, therecently activated drives stay active and the drive which prior to thewrite operation had been active is deactivated.

In yet another alternative embodiment, the switch fromactive-to-inactive or inactive-to-active is controlled by somepredetermined set of procedures. For example, the rules or proceduresmay switch drives in accordance with the number of write operations, inaccordance with the total active time to date, in accordance with amaximum continuous active time, or according to any other predeterminedor adaptive rules. In a preferred embodiment, the rules are selected toapproximately equalize the active or ON time of each drive or drive setwhile restricting the maximum continuous ON time. In this scheme, aninequality of cumulative active time is equalized over a long time.Adjustments to the maximum active time of each different drive or driveset may even be effectuated so as to more rapidly equalized the drivesbut without unduly burdening any drive or drive set. For example, themaximum continuous active time for a disk drive that has a greatercumulative active time may be shortened relative to the maximum activetime permitted for the drive having a lower cumulative active time.These rules and times or other parameters may also be adjusted over thelife of the disk drives to account for aging, remaining capacity, orother operational or physical characteristics.

Thus it will be appreciated that the embodiments of the inventive diskdrive structure and method reduce the power consumption of two or moresets of disc drives or other storage devices, and extend the life cycleof such disc drive or storage devices. The invention also provides acomputer system and method incorporating the power-conserving and devicelife-prolonging system and method described; disc drives susceptible tocontrol in this manner; a disc drive controller for controlling the discdrives or other storage devices in the manner described; computersoftware and/or firmware used in conjunction with the disc drivecontroller and/or computer for controlling the disc drives in the mannerdescribed; as well as a server including such disc drives, controller,software and/or firmware to operate the server in response to read andwrite I/O requests. The invention also provides an operating systemand/or applications program code to implement the inventive disc drivecontrol features described herein.

Additional Embodiments

Having described numerous embodiments of the invention, it will beapparent to those workers having ordinary skill in the applicable artsthat the invention provides a great variety of innovations. Attention isnow directed to highlights of the manner in which selected aspects ofthe invention and innovations may be used either separately or incombination to provide particularly desirable and advantageous utility.Although these highlighted groups of innovations and particularembodiments with each group are particularly useful, the inventions andinnovations described in this specification and the drawings are notlimited only to the embodiments highlighted or otherwise described oridentified below. Within each group of innovations, the selectedembodiments are, for convenience of notation, referred to by embodimentnumbers surrounded by parentheses. These numbers refer to embodimentswithin a particular group of innovations and are reused for thedifferent groups of innovations.

In a first group of innovations, the invention provides variousembodiments associated with System, Method, and Architecture for DynamicServer Power Management and Dynamic Workload Management for Multi-serverEnvironment.

(1) A computer system comprising: a plurality of server computers eachhaving at least one processor and an activity monitor identifying alevel of activity indicator for the at least one processor; each of theserver computers being operable in: (i) a first mode having a firstmaximum performance level and a first power consumption rate. (ii) asecond mode having a second maximum performance level lower than thefirst maximum performance level and a second power consumption ratelower than the first power consumption rate, and (iii) a third modehaving a third maximum performance level lower than the second maximumperformance level and a third power consumption rate lower than thesecond power consumption rate; and a power manager: (i) coupled to eachof the server computers and receiving the level of activity informationfrom each of the plurality of computers; (ii) analyzing the plurality ofreceived level of activity information; (iii) determining an operatingmode for each of the server computers selected from the first mode,second mode, and third mode based on the analyzed activity informationand predetermined policies; and (iv) generating commands to each of theplurality of server computers directing each of the plurality of servercomputers to operate in the determined operating mode.

(2) A computer system comprising: a plurality of computers each havingat least one processor and an activity monitor identifying a level ofactivity indicator for the at least one processor; each of the computersbeing operable in: (i) a first mode having a first maximum performancelevel and a first power consumption rate, and (ii) a third mode having athird maximum performance level lower than the first maximum performancelevel and a third power consumption rate lower than the first powerconsumption rate; and a power manager: (i) coupled to each of thecomputers and receiving the level of activity information from each ofthe plurality of computers; (ii) analyzing the plurality of receivedlevel of activity information; (iii) determining an operating mode foreach of the computers selected from the first mode and third mode basedon the analyzed activity information and predetermined policies; and(iv) generating commands to each of the plurality of computers directingeach of the plurality of computers to operate in the determinedoperating mode.

(3) The computer system in embodiment (2), wherein: each of thecomputers further being operable in (iii) a second mode having a secondmaximum performance level intermediate between the first maximumperformance level and the third maximum performance level and a secondpower consumption rate intermediate between the first power consumptionrate and the third power consumption rate; and the power manager furtherdetermining an operating mode for each of the computers selected fromthe first mode, the second mode, and the third mode based on theanalyzed activity information and the predetermined policies. (4) Thecomputer system in any of embodiments (2 or 3), wherein: the computerscomprise servers. (5) The computer system in any of embodiments (2, 3,or 4), further comprising a power manager computer providing the powermanager. (6) The computer system in any of embodiments (2, 3, or 4)wherein a selected one of the plurality of computers designated as amaster providing the power manager. (7) The computer system in any ofembodiments (2 or 3), wherein the activity monitor comprises an activitymonitor that monitors an activity selected from the set of activitiesconsisting of: a program application layer activity, a network layeractivity, a physical layer activity, and combinations thereof. (8) Asystem as in embodiment (7), wherein at the physical level the number ofprocessor idle threads executed within a predetermined period of timeare measured to determine processor loading and the processorperformance is adjusted to by altering the operating mode tosubstantially match the level of processor loading. (9) The computersystem in embodiment (2), wherein the activity monitor comprises anetwork layer activity monitoring TCP/IP protocol data packets; andprocessor performance is incrementally lowered by the power managerusing the mode control until data packets start dropping indicating thatthe processor performance is at the limit of adequacy and thenincreasing the processor performance by a specified increment to act asa safety margin to provide reliable communication of the packets. (10)The computer system in embodiment (7), wherein the application layeractivity monitor comprises monitoring use of a port address within thecomputers, the monitoring including counting or measuring a number oftimes a specific port address is being requested within a predeterminedperiod of time, and in response to that counting or measurement, placinga sufficient amount of computer performance to meet the performancerequirement for each application requesting the port address. (11) Thecomputer system in embodiment (7), wherein the application layeractivity monitor comprises monitoring use of a port address within thecomputers. (12) The computer system in embodiment (7), wherein thenetwork layer activity monitor comprises monitoring use of a TCP/IPprotocol within the computers. (13) The computer system in embodiment(7), wherein the physical layer activity monitor comprises monitoringthe execution of idle threads within the computers. (14) The computersystem in embodiment (7), wherein the physical layer activity monitorcomprises monitoring counting activities having particular activityvalues within the computers. (15) The computer system in embodiment (3),wherein: the first mode operation is characterized by operating theprocessor at a first processor clock frequency and a first processorcore voltage, the second mode operation is characterized by operatingthe processor at a second processor clock frequency and a secondprocessor core voltage, and the third mode operation is characterized byoperating the processor at a third processor clock frequency and a thirdprocessor core voltage; the second mode of operation being furthercharacterized in that the second processor clock frequency and thesecond processor core voltage in combination consuming less power thanthe first processor clock frequency and the first processor core voltagein combination, and the third processor clock frequency and the thirdprocessor core voltage in combination consuming less power than thesecond processor clock frequency and the second processor core voltagein combination. (16) A system as in embodiment (15), wherein performanceof a group of the computers configured as physical network devicesforming a single logical device are power managed by reducing theperformance and power consumption of each constituent physical device inpredetermined equal increments or predetermined unequal increments. (17)A system as in embodiment (15), wherein network device loading andquality of service (QoS) are measured for a plurality of physicalnetwork devices organized as a single logical network device. (18) Thecomputer system in embodiment (15), wherein the third processor clockfrequency is less than the second processor clock frequency which isless than the first processor clock frequency. (19) The computer systemin embodiment (18), wherein the second processor core voltage is lessthan the first processor core voltage. (20) The computer system inembodiment (19), wherein the third processor core voltage is less thanthe second processor core voltage. (21) The computer system inembodiment (15), wherein the third processor clock frequency is lessthan the second processor clock frequency which is less than the firstprocessor clock frequency; and the second processor core voltage is lessthan the first processor core voltage. (22) The computer system inembodiment (2), wherein: each of the computers further being operable in(iii) a second mode having a second maximum performance levelintermediate between the first maximum performance level and the thirdmaximum performance level and a second power consumption rateintermediate between the first power consumption rate and the thirdpower consumption rate; and each the computer including a local powermanager determining an operating mode for itself selected from the firstmode and the second mode based on processor internal activityinformation. (23) The computer system in embodiment (22), wherein theprocessor internal activity information comprising idle thread executioninformation. (24) The computer system in embodiment (22), wherein atransition from the first mode to the second mode is controlled locallywithin each the computer; and a transition from either the first mode orthe second mode to the third mode are controlled globally by the powermanager. (25) The computer system in embodiment (24), wherein atransition from the second mode to the first mode is controlled locallywithin each the computer; and a transition from the third mode to eitherthe first mode or the second mode is controlled globally by the powermanager. (26) The computer system in embodiment (15), wherein the thirdprocessor clock frequency is substantially zero or the third processorclock is turned off. (27) The computer system in embodiment (15),wherein the commands are generated and directed to the computers onlywhen required to change an operating mode of the computers. (28) Thecomputer system in any of embodiments (2 or 3), wherein the third modeis characterized by maintaining a processor core voltage to maintainprocessor state.

(29) A computer system comprising: a plurality of computers each havingat least one processor and an activity monitor identifying a level ofactivity indicator for the at least one processor; each of the computersbeing operable in: (i) a first mode having a first maximum performancelevel and a first power consumption rate, and (ii) a third mode having athird maximum performance level lower than the first maximum performancelevel and a third power consumption rate lower than the first powerconsumption rate; and a power manager: (i) coupled to each of thecomputers and receiving the level of activity information from each ofthe plurality of computers; (ii) analyzing the plurality of receivedlevel of activity information; (iii) determining an operating mode foreach of the computers selected from the first mode and third mode basedon the analyzed activity information and predetermined policies; and(iv) generating commands to each of the plurality of computers directingeach of the plurality of computers to operate in the determinedoperating mode; each of the computers further being operable in (iii) asecond mode having a second maximum performance level intermediatebetween the first maximum performance level and the third maximumperformance level and a second power consumption rate intermediatebetween the first power consumption rate and the third power consumptionrate; each the computer including a local power manager determining anoperating mode for itself selected from the first mode and the secondmode based on processor internal activity information; a transition fromthe first mode to the second mode is controlled locally within each thecomputer, and a transition from either the first mode or the second modeto the third mode are controlled globally by the power manager; and atransition from the second mode to the first mode is controlled locallywithin each the computer, and a transition from the third mode to eitherthe first mode or the second mode is controlled globally by the powermanager.

(30) A computer system comprising: a plurality of server computers eachhaving at least one processor and an activity monitor identifying alevel of activity for the at least one processor, the activity monitorcomprising an activity monitor that monitors an activity selected fromthe set of activities consisting of: a program application layeractivity, a network layer activity, a physical layer activity, andcombinations thereof; each of the server computers being operable in:(i) a first mode having a first maximum performance level and a firstpower consumption rate, (ii) a second mode having a second maximumperformance level lower than the first maximum performance level and asecond power consumption rate lower than the first power consumptionrate, and (iii) a third mode having a third maximum performance levellower than the second maximum performance level and a third powerconsumption rate lower than the second power consumption rate; and apower manager operative in a separate power manager computer: (i)coupled to each of the server computers and receiving the level ofactivity information from each of the plurality of computers; (ii)analyzing the plurality of received level of activity information; (iii)determining an operating mode for each of the server computers selectedfrom the first mode, second mode, and third mode based on the analyzedactivity information; and (iv) generating commands to each of theplurality of server computers directing each of the plurality of servercomputers to operate in the determined operating mode; the first modeoperation is characterized by operating the processor at a firstprocessor clock frequency and a first processor core voltage, the secondmode operation is characterized by operating the processor at a secondprocessor clock frequency and a second processor core voltage, and thethird mode operation is characterized by operating the processor at athird processor clock frequency and a third processor core voltage; thesecond mode of operation being further characterized in that the secondprocessor clock frequency is lower than the first processor clockfrequency and the second processor core voltage is equal to or less thanthe first processor core voltage so that in combination consuming lesspower than in the first mode, and the third processor clock frequency islower than the second processor clock frequency and the third processorcore voltage is no greater than the second processor core voltage sothat in combination consuming less power than in the second mode; and atransition from the first mode to the second mode is controlled locallywithin each the computer; and a transition from either the first mode orthe second mode to the third mode are controlled globally by the powermanager.

(31) A method of operating computer system having a plurality of servercomputers, each server computer including at least one processor, andeach computer being operable in a first mode having a first maximumperformance level and a first power consumption rate, and a third modehaving a third maximum performance level lower than the first maximumperformance level and a third power consumption rate lower than thefirst power consumption rate; the method comprising: monitoring activitywithin the computers and identifying a level of activity for the atleast one processor within the computers; analyzing the plurality oflevel of activity information; determining an operating mode for each ofthe computers selected from the first mode and third mode based on theanalyzed activity information; and generating commands to each of theplurality of computers directing each of the plurality of computers tooperate in the determined operating mode.

(32) The method in embodiment (31), wherein each of the computersfurther being operable in a second mode having a second maximumperformance level intermediate between the first maximum performancelevel and the third maximum performance level and a second powerconsumption rate intermediate between the first power consumption rateand the third power consumption rate; and the determining an operatingmode further comprising determining an operating mode for each of thecomputers selected from the first mode, the second mode, and the thirdmode based on the analyzed activity information. (33) The method ofembodiment (32), wherein a transition from the first mode to the secondmode is controlled locally within each the computer; and a transitionfrom either the first mode or the second mode to the third mode arecontrolled globally by the power manager. (34) The method of embodiment(33), wherein a transition from the second mode to the first mode iscontrolled locally within each the computer; and a transition from thethird mode to either the first mode or the second mode is controlledglobally by the power manager. (35) A system as in embodiment (15),wherein at least one of a processor clock frequency and a processoroperating voltage is reduced in response to the indicator to therebyreduce power consumed by the processor and by the server computer. (36)A system as in embodiment (15), wherein the processor clock frequency isreduced in response to the indicator to thereby reduce power consumed bythe processor and by the server. (37) A system as in embodiment (15),wherein the indicator comprises a measured decrease in server load. (38)A system as in embodiment (15), wherein the indicator comprises apredicted decrease in server load. (39) A system as in embodiment (15),wherein the indicator comprises a measured decrease in processortasking. (40) A system as in embodiment (15), wherein the indicatorcomprises a predicted decrease in processor tasking. (41) A system as inembodiment (15), wherein the indicator comprises a measured decrease incommunication channel bandwidth. (42) A system as in embodiment (15),wherein the indicator comprises predicted decrease in communicationchannel bandwidth. (43) A system as in embodiment (32), wherein thepredicted decrease in server load is a prediction based at least in parton time of day. (44) A system as in embodiment (32), wherein thepredicted decrease in server load is a prediction based at least in parton a quality of service requirement. (45) A system as in embodiment(32), wherein the predicted decrease in processor tasking is aprediction based at least in part on time of day. (46) A system as inembodiment (32), wherein the predicted decrease in processor tasking isa prediction based at least in part type of content to be processed.(47) A system as in embodiment (32), wherein the predicted decrease inserver loading is a prediction based at least in part type of content tobe served. (48) A system as in embodiment (32), wherein the manner ofthe prediction is further based on the content served by the server.(49) A system as in embodiment (15), wherein the majority of contentserved by the server computer comprises web pages. (50) A system as inembodiment (15), wherein the majority of content served by the servercomputer comprises streaming video. (51) A system as in embodiment (15),wherein the majority of content served by the server computer comprisesmulti-media content. (52) A system as in embodiment (15), wherein themajority of content served by the server comprises cached data. (53) Asystem as in embodiment (15), wherein a processor operating voltage isreduced in response to the indicator to thereby reduce power consumed bythe processor and by the server. (54) A system as in embodiment (15),wherein a predetermined combination of processor core voltage andprocessor frequency are selected based on predetermined rules to satisfya processor load requirement. (55) A system as in embodiment (46),wherein the predetermined rules for selecting the predeterminedcombination of processor core voltage and processor frequency comprise alook-up-table (LUT) identifying processor frequency and processor corevoltage with processor load handling capability. (56) A system as inembodiment (15), wherein each processor has a processor load handlingcapability measured in instructions per second. (57) A system as inembodiment (15), wherein each processor has a processor load handlingcapability is measured in bits served per second. (58) A system as inembodiment (15), wherein each processor has a processor load handlingcapability is measured in transactions per unit time. (59) A system asin embodiment (15), wherein each processor has a processor load handlingcapability is measured in transactions per second. (60) A system as inembodiment (15), wherein the predetermined rules are different fordifferent network device types. (61) A system as in embodiment (3),wherein the predetermined policies include policies for identifying anon-linear relationship between processor performance and powerconsumption. (62) A system as in embodiment (3), wherein power (P)consumed by a circuit in the processor is proportional to a capacitance(C) times the product of the switching frequency of the circuit (f) andthe square of the circuit operating voltage (v²) or P=Cfv². (63) Asystem as in embodiment (52), wherein switching frequency issubstantially linear relative to power consumption, the load isnon-linear relative to circuit performance, and the load is non-linearrelative to power consumed by the circuit. (64) A system as inembodiment (53), wherein the circuit comprises a processor and randomaccess memory. (65) A system as in embodiment (3), wherein thepredetermined policy includes a policy for identifying a substantiallylinear relationship between processor performance and power consumption.(66) A system as in embodiment (3), wherein the processor providesprocessing for a web server and the web server has a substantiallylinear relationship between web page server loading an power consumed inserving the web pages. (67) A system as in embodiment (3), wherein aquality-of-service (QoS) is first established, and a processorperformance is established based on predetermined policies that select aprocessor clock frequency, and a minimum processor core voltage isselected to match the selected processor clock frequency; and whereinthe established processor performance is used to control an operatingmode. (68) A system as in embodiment (15); further including a processorcore voltage control circuit receiving voltage control signals andincreasing or decreasing the processor core voltage in response to thereceipt. (69) A system as in embodiment (58), wherein the processor corevoltage control circuit provides a direct-current voltage to a Vccsupply terminal of the processor. (70) A system as in embodiment (15),wherein the activity level indicator includes an indicator of the numberof idle threads executed in the processor, and reduction of processorpower consumption is initiated based on detection of the execution of anidle thread by the processor. (71) A system as in embodiment (60),wherein upon detection of execution of the idle thread, the processorfrequency is reduced as compared to a maximum processor clock frequency.(72) A system as in embodiment (61), wherein the processor frequencyreduction is a reduction by a factor of a power of two. (73) A system asin embodiment (61), wherein the processor clock frequency is reduced tozero. (74) A system as in embodiment (61), wherein the processorfrequency is reduced to an integral multiple of a maximum processorclock frequency. (75) A system as in embodiment (60), wherein upondetection of execution of the idle thread, the processor frequency isreduced as compared to a maximum processor clock frequency and theprocessor core voltage is reduced as compared to a maximum processorcore voltage. (76) A system as in embodiment (60), wherein the detectionof execution of an idle thread initiated power reduction provides a realtime adjustment to power consumption based on measured processor loadrequirements. (77) A system as in embodiment (57), wherein the QoSinitiated power reduction provides a preset adjustment to powerconsumption based on predicted processor load requirements. (78) Asystem as in embodiment (57), wherein the QoS requirement is adjusted onat least one of a time-of-day criteria, a day-of-week criteria, aseasonal criteria, and combinations thereof. (79) A system as inembodiment (57), wherein the QoS requirement is adjusted based oncriteria selected from the set consisting of: time-of-day, day-of-month,day-of week, month-of year, geographic location of requester, requesteridentity, requester account number, and combinations thereof (80) Thecomputer system in embodiment (3), wherein the activity indicatorcomprises a network quality of service indicator. (81) A system as inembodiment (3), wherein power is conserved by controlling each computernode to enter one of the second mode or the third mode using one or moreof a quality of service based predictive processor performance reductionand a activity based measured performance requirement. (82) A system asin embodiment (3), wherein when there is a requirement that one computerbe placed in a lower power consumption mode, the computer selected forsuch lower power consumption is selected according to predeterminedrules such that different computers are placed in lower powerconsumption mode each time such selection is required. (83) A system asin embodiment (3), wherein a computer placed in mode 3 is in a suspendstate and may be woken up and placed in the first mode or the secondmode by any one of a plurality of events including by a wake on LANsignal event. (84) A system as in embodiment (3), wherein the transitionfrom one power consumption mode to another power consumption mode isbased on a procedure implemented in software. (85) A system as inembodiment (3), wherein the transition from one power consumption modeto another power consumption mode is based on a procedure implemented inhardware and software. (86) A system as in embodiment (3), wherein whenthere is need to operate fewer than all the computer, the particularcomputer or logical group of computers that is (are) turned off orplaced in a reduced power consumption mode is cycled so that over timeall of the network devices experience similar operating time histories.(87) A system as in embodiment (3), wherein at least some of thecomputers include a mass storage device including a rotatable storagedevice. (88) A system as in embodiment (3), wherein the computers areconfigured as network server devices and a network load versus allocatednetwork device performance profile is provided for each different typeof network server device, and the performance level set for operation ofthe network device is established by reference to the profile. (89) Asystem as in embodiment (3), wherein at least one of the computerscomprises a network server device and the activity monitoring for thenetwork server device comprises a monitoring or either the networkdevice load or the network device quality of service (QoS); and whereinthe monitoring is performed by the activity monitor or by a separatemanagement computer, or both. (90) A system as in embodiment (3),wherein the system includes at least one temperature sensor within anenclosure holding the computers for monitoring and reporting thetemperature proximate the sensor to a computers configured to monitorthe temperature. (91 ) A system as in embodiment (3), wherein the systemincludes a plurality of temperature sensors within the enclosurereporting to one or more network devices. (92) A system as in embodiment(3), wherein a plurality of cooling fans are provided and operate undercontrol of the power manager that controls each fan to provide coolingat the rate and location desired to maintain the computers within apredetermined temperature range. (93) A system as in embodiment (3),wherein the plurality of computers are disposed within a commonenclosure and the system further comprising a plurality of temperaturesensors and a plurality of cooling devices are also disposed within theenclosure, the plurality of temperature sensors communicating atemperature signal to a temperature control means and the control meansadjusting the on/off status and operational parameters of the coolingunits to extract heat according to predetermined rules. (94) A system asin embodiment (2), wherein the system further includes a plurality ofpower supplies and the power supplies are controlled to maintain arequired power output level and operate the power supplies at apreferred efficiency.

(95) A system as in embodiment (3), wherein the temperature of thesystem is moderated by motor driven cooling fans and wherein arotational speed of the motor drive cooling is adjusted to maintain apredetermined temperature range proximate a temperature sensor. (96) Asystem as in embodiment (3), wherein the rotational speed of a motordrive cooling is adjusted to maintain a predetermined temperature rangewithin an enclosure. (97) A system as in embodiment (15), wherein theactivity level indicator includes an indicator of the number of idlethreads executed in the processor. (98) A system as in embodiment (60),wherein upon detection of execution of the idle thread, the processorclock frequency is adjusted in real time so that the capability of theprocessor is substantially matched to the required processingcapability. (99) A system as in embodiment (101), wherein the processorclock frequency is adjusted so that no idle threads, a predeterminednumber of idle threads, or a predetermined occurrence frequency of idlethreads result.

(100) A system as in embodiment (57), wherein the QoS requirement isadjusted based on criteria selected from the set consisting of:time-of-day, day-of-month, day-of week, month-of year, geographiclocation of requester, requester identity, requester account number, andcombinations thereof. (101) A system as in embodiment (3), wherein whenthe system includes a plurality of network devices and there is arequirement that one network device be placed in a lower powerconsumption mode, the network device selected for such lower powerconsumption is selected according to predetermined policies such thatdifferent network devices are placed in lower power consumption modeeach time such selection is required. (102) A system as in embodiment(101), wherein the predetermined policies provide for random selectionof one of the network devices. (103) A system as in embodiment (101),wherein the predetermined policies provide for cycling through thenetwork devices according to some predetermined ordering. (104) A systemas in embodiment (101), wherein the predetermined policies provide forcycling through the network devices according to some predeterminedordering in which network devices having the lowest time in service arepreferentially selected for continued operation and network deviceshaving the longest time in service are selected for reduced poweroperation. (105) A system as in embodiment (101), wherein the reducedpower operation includes being powered off. (106) A system as inembodiment (101), wherein the reduced power operation includes beingplaced in a suspend mode. (107) A system as in embodiment (101), whereinthe reduced power operation is determined according to a procedure forcontrolling power consumption by the system, the system having aplurality of computers operating as the network devices, each computerincluding at least one processor, and each computer being operable in afirst mode having a first maximum performance level and a first powerconsumption rate, and a third mode having a third maximum performancelevel lower than the first maximum performance level and a third powerconsumption rate lower than the first power consumption rate; theprocedure comprising: monitoring activity within the computers andidentifying a level of activity for the at least one processor withinthe computers; analyzing the plurality of level of activity information;determining an operating mode for each of the computers selected fromthe first mode and third mode based on the analyzed activityinformation; and generating commands to each of the plurality ofcomputers directing each of the plurality of computers to operate in thedetermined operating mode. (108) The system in embodiment (107), whereineach of the computers further being operable in a second mode having asecond maximum performance level intermediate between the first maximumperformance level and the third maximum performance level and a secondpower consumption rate intermediate between the first power consumptionrate and the third power consumption rate; and the determining anoperating mode further comprising determining an operating mode for eachof the computers selected from the first mode, the second mode, and thethird mode based on the analyzed activity information. (109) The systemof embodiment (108), wherein a transition from the first mode to thesecond mode is controlled locally within each the computer; and atransition from either the first mode or the second mode to the thirdmode are controlled globally by the power manager. (110) The system ofembodiment (109), wherein a transition from the second mode to the firstmode is controlled locally within each the computer; and a transitionfrom the third mode to either the first mode or the second mode iscontrolled globally by the power manager.

(111) In a server farm comprising a multiplicity of computer systemsoperating as content servers, a method of operating the servers, eachserver computer including at least one processor, and each computerbeing operable in a first mode having a first maximum performance leveland a first power consumption rate, and a third mode having a thirdmaximum performance level lower than the first maximum performance leveland a third power consumption rate lower than the first powerconsumption rate; the method comprising: monitoring activity within eachthe computer server and identifying a level of activity for the at leastone processor within the server computer; analyzing the plurality oflevel of activity information; determining an operating mode for each ofthe computers selected from the first mode and third mode based on theanalyzed activity information; and generating commands to each of themultiplicity of server computers directing each of the plurality ofcomputers to operate in the determined operating mode.

In a second group of innovations, the invention provides variousembodiments associated with System and Method for Activity or EventBased Dynamic Energy Conserving Server Reconfiguration.

(1) An information processing system comprising: a frame or enclosurefor mounting a plurality of devices; a backplane having a plurality ofbackplane electrical connectors disposed within the frame or enclosure;and a plurality of devices, each including a device electricalconnector, matingly coupled to the backplane electrical connectors, theplurality of devices including at least one network device for couplingthe system with an external network.

(2) A system as in embodiment (1), wherein the at least one networkdevice comprises a device selected from the set of network devicesconsisting of a server device, a computer node device, a monitor nodedevice, a management module, a server module, and combinations thereof.(3) A system as in embodiment (2), wherein the at least one networkdevice includes a processor and a memory integral with or coupled to theprocessor. (4) A system as in embodiment (3), further comprising anetwork switch or network switching device. (5) A system as inembodiment (4), wherein the plurality of devices further comprises adevice selected from the set of devices consisting of a power supply, afan or fan module, and combinations thereof. (6) A system as inembodiment (1), wherein the at least one network device comprises atleast one server computer having at least one processor and a powermanager. (7) A system as in embodiment (6), wherein the power manager isintegral with the server computer. (8) A system as in embodiment (6),wherein the power manager is separate from the server computer. (9) Asystem as in embodiment (6), wherein the or each server computer furthercomprises an activity monitor identifying a level of activity indicatorfor the at least one processor; and the or each server computer beingoperable in: (i) a first mode having a first maximum performance leveland a first power consumption rate, and (ii) a third mode having a thirdmaximum performance level lower than the first maximum performance leveland a third power consumption rate lower than the first powerconsumption rate; and the system further comprising: a power manager:(i) coupled to each of the computers and receiving the level of activityinformation from each of the plurality of computers; (ii) analyzing theplurality of received level of activity information; (iii) determiningan operating mode for each of the computers selected from the first modeand third mode based on the analyzed activity information andpredetermined policies; and (iv) generating commands to each of theplurality of computers directing each of the plurality of computers tooperate in the determined operating mode. (10) A system as in embodiment(9), wherein: the or each server computer further being operable in:(iii) a second mode having a second maximum performance levelintermediate between the first maximum performance level and the thirdmaximum performance level and a second power consumption rateintermediate between the first power consumption rate and the thirdpower consumption rate; and the power manager further determining anoperating mode for each of the computers selected from the first mode,the second mode, and the third mode based on the analyzed activityinformation and the predetermined policies. (11) A system as inembodiment (1), wherein the system further comprises a power manager.(12) A system as in embodiment (10), wherein the system furthercomprises a switching module, and the power manager receives activityindicators for the switching module and controls an operating mode ofthe switching module in response thereto. (13) A system as in embodiment(10), wherein the computer comprises a server module that is powermanaged by adjusting processor performance to one or more of a predictedprocessor processing requirement and a measured processor processingrequirement. (14) A system as in embodiment (13), wherein the predictedprocessor processing requirement is a Quality of Service (QoS) basedrequirement, and the measured processor processing requirement comprisesa substantially real-time measured processor processing requirement.(15) A system as in embodiment (14), wherein the substantially real-timeprocessor processing requirement comprises an idle thread executiondetection and response thereto. (16) A system as in embodiment (10),wherein power (or energy) is conserved by controlling the computer basedon a control procedure algorithm to enter a first level of power(energy) saving by adjusting the performance of the processor within thecomputer to substantially match the computer processor loading demand.(17) A system as in embodiment (10), wherein power (or energy) isconserved by controlling the plurality of computers in aggregate basedon a control procedure algorithm and the policy to enter selected levelsof power (energy) saving by adjusting the performance of the processorswithin the computers to one of the first mode, second mode, and thirdmode to substantially match the aggregate computer processor loadingdemands. (18) A system as in embodiment (10), wherein the power managerincludes a control procedure algorithm implemented as software toimplement a power on demand control procedure. (19) A system as inembodiment (10), wherein each computer is configurable as a particulartype of network device. (20) A system as in embodiment (10), wherein thecomputer is configured as a network device selected from the setconsisting of a web server, a streaming media server, a cache server, afile server, an application server, and a router. (21) A system as inembodiment (10), wherein at least selected ones of the computers areconfigurable as a combination type of network device, and wherein thenetwork device configured in the computer node is a network deviceselected from the set consisting of a web server, a streaming mediaserver, a cache server, a file server, an application server, a router,and combinations thereof. (22) A system as in embodiment (21), whereinthe network device is reconfigurable at any time based on types ofactivities detected within the network to which the network device is ormay be connected. (23) A system as in embodiment (10), wherein at leastone of the computers comprises a network device and the activity monitorfor the network device comprises a network activity monitor that detectsthe types of activities present on a network to which the activitymonitor is coupled. (24) A system as in embodiment (23), wherein thetypes of activities present on a network to which the activity monitoris coupled that are monitored by the activity monitor include volume ofweb pages served, volume of streaming media served, volume of filesserved, volume of applications served, volume of cached data served,amount of network traffic routed, and combinations thereof (25) A systemas in embodiment (22), wherein the reconfiguration of network device isinitiated by any network device including the same network as is beingreconfigured. (26) A system in embodiment (4) wherein a selected one ofthe plurality of computers designated as a master providing the powermanager. (27) A system as in embodiment (10), wherein a selected one ofthe plurality of computers is designated as a master providing the powermanager, and reconfiguration of a network device from one form ofnetwork device to another form of network device is initiated by anycomputer that has been designated as a master computer. (28) A system asin embodiment (27), wherein any computer may be designated as the masternode. (29) A system as in embodiment (28), wherein a particular computeris designated as a master on the basis of its position within a chassis.(30) A system as in embodiment (28), wherein a particular computer nodeis designated as a master node on the basis of the order of power-up orboot completion. (31) A system as in embodiment (28), whereinreconfiguration of the computer comprises altering the software and/orfirmware instructing the computer. (32) A system as in embodiment (28),wherein reconfiguration of the computer comprises altering the dataorganization of a data storage device integral with or coupled to thecomputer. (33) A system as in embodiment (28), wherein the data storagedevice comprises a hard disc drive based RAID storage array and alteringthe data organization comprises altering rad configuration of the datato provide better performance for the type of data being served. (34) Asystem as in embodiment (22), wherein the reconfiguration of a computeris initiated by a management module network device. (35) A system as inembodiment (10), wherein a plurality of computers of the same type aregrouped together and treated as a single network device. (36) A systemas in embodiment (35), wherein the group of network devices treated as asingle network device is managed and controlled as a single networkdevice. (37) A system as in embodiment (35), wherein the group ofnetwork devices treated as a single network device is power managed as asingle network device. (38) A system as in embodiment (35), wherein thegroup of network devices treated as a single network device is monitoredas a single network device. (39) A system as in embodiment (35), whereinthe plurality of grouped network devices are electrically coupled via abackplane bus and the logical grouping of the plurality of networkdevices into a single logical network device is performed under controlof software. (40) A system as in embodiment (39), wherein the softwareexecutes within a processor and memory associated within each networkdevice. (41) A system as in embodiment (35), wherein the plurality ofnetwork devices each comprise a server group. (42) A system as inembodiment (35), wherein the plurality of network devices each comprisea computer server module. (43) A system as in embodiment (42), whereineach computer server module is configured as a computer server moduleselected from the group consisting of a web server, a streaming mediaserver, a cache server, a file server, an application server, a router,and combinations thereof (44) A system as in embodiment (39), whereinthe activity associated with each computer within a grouped logicalnetwork device may be monitored individually. (45) A system as inembodiment (39), wherein the network activity associated with all or anysubset of physical network device within a grouped logical networkdevice may be monitored as a composite or in aggregate. (46) A system asin embodiment (35), wherein grouping is accomplished by aggregating allof the activity in each computer and directing each computer in thelogical group to operate at the same operating mode. (47) A system as inembodiment (10), wherein over a period of time the system will havesufficient over capacity that some of the computers will be directed tooperate in the third mode, the policy taking into account the amounteach of the computers have historically spent operating in at least oneof the first, second, or third mode and selecting a computer to operatein the third mode based on historical data. (48) A system as inembodiment (47), wherein the computer selected to operate in the thirdmode is a computer that has the smallest cumulative duration operatingin the third mode amongst the plurality of computers. (49) A system asin embodiment (47), wherein the computer selected to operate in thethird mode is randomly selected from amongst the plurality of computers.(50) A system as in embodiment (47), wherein the computer selected tooperate in the third mode is rotated sequentially amongst the pluralityof computers. (51) A system as in embodiment (10), wherein the activitymonitor comprises an activity monitor that monitors an activity selectedfrom the set of activities consisting of: a program application layeractivity, a network layer activity, a physical layer activity, andcombinations thereof. (52) A system as in embodiment (51), wherein atthe physical level the number of processor idle threads executed withina predetermined period of time are measured to determine processorloading and the processor performance is adjusted to by altering theoperating mode to substantially match the level of processor loading.(53) A system as in embodiment (52), wherein the substantial matching ofprocessor performance to processor loading is performed with apredetermined amount of additional processor performance beyond thatneeded to match the processor loading. (54) A system as in embodiment(53), wherein the predetermined amount of additional processorperformance is between about one-percent and about five-percentadditional performance. (55) The computer system in embodiment (10),wherein: the first mode operation is characterized by operating theprocessor at a first processor clock frequency and a first processorcore voltage, the second mode operation is characterized by operatingthe processor at a second processor clock frequency and a secondprocessor core voltage, and the third mode operation is characterized byoperating the processor at a third processor clock frequency and a thirdprocessor core voltage; the second mode of operation being furthercharacterized in that the second processor clock frequency and thesecond processor core voltage in combination consuming less power thanthe first processor clock frequency and the first processor core voltagein combination, and the third processor clock frequency and the thirdprocessor core voltage in combination consuming less power than thesecond processor clock frequency and the second processor core voltagein combination. (56) A system as in embodiment (55), wherein performanceof a group of the computers configured as physical network devicesforming a single logical device are power managed by reducing theperformance and power consumption of each constituent physical device inpredetermined equal increments or predetermined unequal increments. (57)A system as in embodiment (56), wherein the unequal increments includeplacing one or more of the plurality of physical devices in the thirdmode operating mode. (58) A system as in embodiment (56), wherein theunequal increments include placing one or more of the plurality ofphysical devices in the second mode operating mode. (59) A system as inembodiment (56), wherein the unequal increments include placing one ormore of the plurality of physical devices in a powered-off fourth mode.

(60) A system as in embodiment (56), wherein a composite performance ofa logical network device is achieved by placing some physical networkdevices in the second mode and by placing others in a different mode.(61) The computer system in embodiment (8), wherein the activity monitorcomprises a network layer activity monitoring TCP/IP protocol datapackets; and processor performance is incrementally lowered by the powermanager using the mode control until data packets start droppingindicating that the processor performance is at the limit of adequacyand then increasing the processor performance by a specified incrementto act as a safety margin to provide reliable communication of thepackets. (62) A system as in embodiment (61), wherein the specifiedincrement is a one-percent to five percent increment. (63) A system asin embodiment (61), wherein the specified increment is a 0.1 percent to10 percent increment. (64) The computer system in embodiment (9),wherein the activity monitor comprises an activity monitor that monitorsan activity selected from the set of activities consisting of: a programapplication layer activity, a network layer activity, a physical layeractivity, and combinations thereof. (65) The computer system inembodiment (64), wherein the application layer activity monitorcomprises monitoring use of a port address within the computers, themonitoring including counting or measuring a number of times a specificport address is being requested within a predetermined period of time,and in response to that counting or measurement, placing a sufficientamount of computer performance to meet the performance requirement foreach application requesting the port address. (66) A system as inembodiment (65), wherein the sufficient amount of network performance isprovided by operating selected computer in a first predeterminedperformance having a predetermined power consumption and a second groupof other selected physical network devices at a reduced secondperformance level having a power consumption lower than that of thefirst selected group. (67) A system as in embodiment (66), wherein thefirst predetermined performance is a maximum performance and the secondpredetermined performance is a second level power saving mode. (68) Asystem as in embodiment (66), wherein the first predeterminedperformance is a maximum performance and the second predeterminedperformance is a third level power saving mode. (69) A system as inembodiment (65), wherein the measurement is determined via a SNMP agent.(70) A system as in embodiment (9), wherein the power manager appliesdifferent policies for different application types including usingdifferent rules to determine and predict system performancerequirements. (71) A system as in embodiment (70), wherein the differentapplication types comprise different server types. (72) A system as inembodiment (70), wherein the different rules comprise differentmeasurement procedures. (73) A system as in embodiment (70), wherein thesystem performance requirements comprise processor performancerequirements. (74) A system as in embodiment (70), wherein the systemperformance requirements comprise server loading performancerequirements. (75) A system as in embodiment (70), wherein theapplication type comprises a network application. (76) A system as inembodiment (75), wherein the network application comprises a networkfile server (NFS) application. (77) The system in embodiment (76),wherein the computer comprises a network server, and a processor withinthe computer operates at a processor clock frequency just sufficient tomaintain maximum rated communication over a predetermined networkconnection. (78) The system in embodiment (77), wherein the apredetermined network connection comprises a 100 Mbps ethernetconnection. (79) A system as in embodiment (77), wherein the processorclock frequency is less than about 300 MHz. (80) A system as inembodiment (75), wherein the processor clock frequency is less thanabout 300 MHz. (81) The computer system in embodiment (10), wherein theactivity indicator comprises a network quality of service indicator.(82) A system as in embodiment (10), wherein power is conserved bycontrolling each computer node to enter one of the second mode or thethird mode using one or more of a quality of service based predictiveprocessor performance reduction and a activity based measuredperformance requirement. (83) A system as in embodiment (82), whereinthe activity based measured performance comprises an idle threadexecution based activity measure. (84) A system as in embodiment (81),wherein a plurality of the computers are organized as a single logicalnetwork device, and network device loading and QoS are measured forlogical network device. (85) A system as in embodiment (81), whereinwithin the single logical network device, at least some computers makingup the logical network device enter the third mode while other of thephysical network devices operate in one or more of the first and secondmodes. (86) A system as in embodiment (81), wherein the computers canenter a third mode directly or indirectly from either the first mode orthe second mode. (87) A system as in embodiment (10), wherein when thereis a requirement that one computer be placed in a lower powerconsumption mode, the computer selected for such lower power consumptionis selected according to predetermined rules such that differentcomputers are placed in lower power consumption mode each time suchselection is required. (88) A system as in embodiment (87), wherein thepredetermined rules provide for random selection of one of thecomputers. (89) A system as in embodiment (87), wherein thepredetermined rules provide for cycling through the computers accordingto some predetermined ordering. (90) A system as in embodiment (89),wherein the predetermined rules provide for cycling through thecomputers according to some predetermined ordering in which computershaving the lowest time in service are preferentially selected forcontinued operation and network devices having the longest time inservice are selected for reduced power operation. (91) A system as inembodiment (90), wherein the reduced power operation includes beingpowered off. (92) A system as in embodiment (90), wherein the reducedpower operation includes being placed in a suspend mode. (93) A systemas in embodiment (10), wherein a computer placed in mode 3 is in asuspend state and may be woken up and placed in the first mode or thesecond mode by any one of a plurality of events including by a wake onLAN signal event. (94) A system as in embodiment (10), wherein thetransition from one power consumption mode to another power consumptionmode is based on a procedure implemented in software. (95) A system asin embodiment (10), wherein the transition from one power consumptionmode to another power consumption mode is based on a procedureimplemented in hardware and software. (96) A system as in embodiment(10), wherein when there is need to operate fewer than all the computer,the particular computer or logical group of computers that is (are)turned off or placed in a reduced power consumption mode is cycled sothat over time all of the network devices experience similar operatingtime histories. (97) A system as in embodiment (96), wherein thecomputers include a non-volatile memory for storing operational history.(98) A system as in embodiment (97), wherein the operational historyincludes a total operating time indicator. (99) A system as inembodiment (97), wherein the operational history includes a time inservice indicator. (100) A system as in embodiment (97), wherein theoperational history includes indicators for operational time at eachoperational mode. ( 101 ) A system as in embodiment (10), wherein atleast some of the computers include a mass storage device including arotatable storage device. (102) A system as in embodiment (101), whereinthe rotatable mass storage device comprises a rotatable magnetic harddisk drive. (103) A system as in embodiment (101), wherein the rotatablemass storage device comprises a rotatable optical disk drive. (104) Asystem as in embodiment (101), wherein the rotatable mass storage devicecomprises a rotatable magneto-optical disk drive. (105) A system as inembodiment (101), wherein the rotatable mass storage device is powermanaged by controlling the rotation of a motor rotating the rotatabledevice, wherein the disc drive is not rotated when a computer associatedwith the drive is in a mode 3 operating condition. (106) A system as inembodiment (10), wherein the computers are configured as network serverdevices and a network load versus allocated network device performanceprofile is provided for each different type of network server device,and the performance level set for operation of the network device isestablished by reference to the profile. (107) A system as in embodiment(106), wherein the profile is implemented as an analytical expressionexecuted in software or firmware. (108) A system as in embodiment (106),wherein the profile is implemented as a piecewise linear expressionexecuted in software or firmware. (109) A system as in embodiment (106),wherein the profile is implemented as a look-up-table stored in amemory. (110) A system as in embodiment (10), wherein at least one ofthe computers comprises a network server device and the activitymonitoring for the network server device comprises a monitoring oreither the network device load or the network device quality of service(QoS); and wherein the monitoring is performed by the activity monitoror by a separate management computer, or both. (111) A system as inembodiment (10), wherein the system includes at least one temperaturesensor within an enclosure holding the computers for monitoring andreporting the temperature proximate the sensor to a computers configuredto monitor the temperature. (112) A system as in embodiment (10),wherein the system includes a plurality of temperature sensors withinthe enclosure reporting to one or more network devices. (113) A systemas in embodiment (112), wherein the plurality of temperature sensors arespatially distributed to provide temperature monitoring of differentnetwork devices within the enclosure. (114) A system as in embodiment(112), wherein the plurality of temperature sensors are spatiallydistributed to provide temperature monitoring of different networkdevices and power supplies within the enclosure. (115) A system as inembodiment (111), wherein when the temperature sensed by a temperaturesensor is within a predetermined magnitude relationship of a firstpredetermined value at least one computer is transitioned to a lowerpower consumption state. (116) A system as in embodiment (115), whereinwhen the temperature sensed by a temperature sensor is within apredetermined magnitude relationship of a second predetermined value atleast one computer is transitioned to a powered off state. (117) Asystem as in embodiment (111), wherein the operational mode of at leastone computer is reduced to a lower power consuming and heat dissipatingstate in response to a temperature sensor reporting a temperaturegreater than or equal to a predetermined value. (118) A system as inembodiment (111), wherein after the power consumption operating mode hasbeen lowered permitting the computer to be operated at a higher powerconsuming state when the temperature sensed is below a predeterminedtemperature value, the lower temperature value being selected to providehysteresis and prevent oscillation between higher power state and lowerpowered state. (119) A system as in embodiment (115), wherein the lowerpower consumption state is achieved by lowering the clock frequency ofthe processor, the clock frequency of a bus coupling a processor toother components, or the operating voltage of the processor or othercomponents. (120) A system as in embodiment (115), wherein theparticular network device that is transitioned to a lower powerconsumption state is selected based on predetermined rules. (121) Asystem as in embodiment (120), wherein the predetermined rules include aquality of service indicator. (122) A system as in embodiment (121),wherein additional computer devices are sent to lower energy consumingmodes if the temperature remains above a predetermined temperaturevalue. (123) A system as in embodiment (10), wherein power consumptionwithin the system is reduced by adjusting the number and motor speed ofcooling fans responsible for cooling the computer. (124) A system as inembodiment (10), wherein a plurality of cooling fans are provided andoperate under control of the power manager that controls each fan toprovide cooling at the rate and location desired to maintain thecomputers within a predetermined temperature range. (125) A system as inembodiment (10), wherein the plurality of computers are disposed withina common enclosure and the system further comprising a plurality oftemperature sensors and a plurality of cooling devices are also disposedwithin the enclosure, the plurality of temperature sensors communicatinga temperature signal to a temperature control means and the controlmeans adjusting the on/off status and operational parameters of thecooling units to extract heat according to predetermined rules. (126) Asystem as in embodiment (125), wherein the power manager comprises thetemperature control means. (127) A system as in embodiment (125),wherein one of the computers within the enclosure comprises thetemperature control means. (128) A system as in embodiment (9), whereinthe system further includes a plurality of power supplies and the powersupplies are controlled to maintain a required power output level andoperate the power supplies at a preferred efficiency. (129) A system asin embodiment (128), wherein only selected ones of the plurality ofpower supplies are operated. (130) A system as in embodiment (128),wherein multiple ones of the power supplies are operated but each isoperated at less than rated power output capacity. (131) A system as inembodiment (10), wherein the temperature of the system is moderated bymotor driven cooling fans and wherein a rotational speed of the motordrive cooling is adjusted to maintain a predetermined temperature rangeproximate a temperature sensor. (132) A system as in embodiment (10),wherein the rotational speed of a motor drive cooling is adjusted tomaintain a predetermined temperature range within an enclosure.

(133) A power-conservative multi-node network device, comprising: anenclosure having a power supply and a back-plane bus; a plurality ofhot-pluggable node devices in the form of printed circuit (PC) cardsadapted for connection with the back-plane buss; and each the nodedevice being reconfigurable in substantially real-time to adapt tochanging conditions on the network.

(134) The network device in embodiment (133), wherein the plurality ofhot-pluggable node devices comprise up to sixteen node devices. (135)The network device in embodiment (133), wherein each of the node devicesincludes power saving control features.

(136) A computer program product for use in conjunction with a computersystem having a plurality of server computers, each server computerincluding at least one processor, and each computer being operable in afirst mode having a first maximum performance level and a first powerconsumption rate, and a third mode having a third maximum performancelevel lower than the first maximum performance level and a third powerconsumption rate lower than the first power consumption rate, thecomputer program product comprising a computer readable storage mediumand a computer program mechanism embedded therein, the computer programmechanism, comprising: a program module that directs at least onecomputer, to function in a specified manner, the program moduleincluding instructions for: monitoring activity within the computers andidentifying a level of activity for the at least one processor withinthe computers; analyzing the plurality of level of activity information;determining an operating mode for each of the computers selected fromthe first mode and third mode based on the analyzed activityinformation; and generating commands to each of the plurality ofcomputers directing each of the plurality of computers to operate in thedetermined operating mode.

(137) The computer program product of embodiment (136), wherein each ofthe computers further being operable in a second mode having a secondmaximum performance level intermediate between the first maximumperformance level and the third maximum performance level and a secondpower consumption rate intermediate between the first power consumptionrate and the third power consumption rate; and the determining anoperating mode further comprising determining an operating mode for eachof the computers selected from the first mode, the second mode, and thethird mode based on the analyzed activity information. (138) Thecomputer program product of embodiment (137), wherein a transition fromthe first mode to the second mode is controlled locally within each thecomputer; and a transition from either the first mode or the second modeto the third mode are controlled globally by the power manager. (139)The computer program product of embodiment (138), wherein a transitionfrom the second mode to the first mode is controlled locally within eachthe computer; and a transition from the third mode to either the firstmode or the second mode is controlled globally by the power manager.

In a third group of innovations, the invention provides variousembodiments associated with System, Method, Architecture, and ComputerProgram Product for Dynamic Power Management in a Computer System.

(1) In a computer system including at least one processing unit, amemory coupled to the at least one processing unit, and logic circuitscoupled to the processing unit contributing to operation of the computersystem, a method for controlling the operating mode and as a result thepower consumption of the computer system between a plurality ofoperating modes each having a different electrical power consumptionlevels or ranges; the method comprising: while operating in a firstselected operating mode exhibiting that first selected mode'scharacteristic power consumption range, (i) monitoring the computersystem to detect the occurrence or non-occurrence of a first event; and(ii) transitioning the computer system from the first selected operatingmode to a second selected operating mode exhibiting that second selectedoperating mode's power consumption range.

(2) The method in embodiment (1), wherein the first selected mode is ahigher power consuming mode than the second selected mode. (3) Themethod in embodiment (1), wherein the first selected mode is a lowerpower consuming mode than the second selected mode. (4) The method inembodiment (1), wherein the computer system further comprises peripheraldevices coupled to the at least one processing unit and the peripheraldevices are power managed to reduce power consumption. (5) The method inembodiment (4), wherein the peripheral devices include a mass storagedevice storing data for retrieval of the data, and an output port foroutputting selected portions of the stored data upon request. (6) Themethod in embodiment (1), wherein the first event comprises execution ofa predetermined number of idle threads. (7) The method in embodiment(1), wherein the first event comprises execution of a single idlethread. (8) The method in embodiment (1), wherein the first eventcomprises execution of a predetermined plurality of idle threads. (9)The method in embodiment (1), wherein the first event comprises a wakeon LAN signal event. (10) The method in embodiment (1), wherein thefirst event comprises the occurrence of some specified level of CPUprocessing capability availability that is derived from either anenumeration or a statistical evaluation of the idle thread or idlethreads that are being or have been executed during some time period.(11) The method in embodiment (1), wherein one of the first and secondevents comprises a measured decrease in server load. (12) The method inembodiment (1), wherein one of the first and second events comprises apredicted decrease in server load. (13) The method in embodiment (1),wherein one of the first and second events comprises a measured decreasein processor tasking. (14) The method in embodiment (1), wherein one ofthe first and second events comprises a predicted decrease in processortasking. (15) The method in embodiment (1), wherein one of the first andsecond events comprises a measured decrease in communication channelbandwidth. (16) The method in embodiment (1), wherein one of the firstand second events comprises predicted decrease in communication channelbandwidth. (17) The method in embodiment (12), wherein the predicteddecrease in server load is a prediction based at least in part on timeof day. (18) The method in embodiment (12), wherein the predicteddecrease in server load is a prediction based at least in part on aquality of service requirement. (19) The method in embodiment (12),wherein the predicted decrease in processor tasking is a predictionbased at least in part on time of day. (20) The method in embodiment(12), wherein the predicted decrease in processor tasking is aprediction based at least in part type of content to be processed by thecomputer system. (21) The method in embodiment (12), wherein thepredicted decrease in server loading is a prediction based at least inpart type of content to be served by the computer system. (22) Themethod in embodiment (12), wherein the manner of the prediction isfurther based on the content served by the server computer system. (23)The method in embodiment (1), wherein one of the first selectedoperating mode and the second selected operating mode comprises a mode(Mode 1) in which the processing unit is operated at substantiallymaximum rated processing unit clock frequency and at substantiallymaximum rated processing unit core voltage, and the logic circuit isoperated at substantially maximum rated logic circuit clock frequencyand at a substantially maximum rated logic circuit operating voltage.(24) The method in embodiment (1), wherein one of the first selectedoperating mode and the second selected operating mode comprises a mode(Mode 2) in which the processing unit is operated at less than maximumrated processing unit clock frequency and at less than or equal to amaximum rated processing unit core voltage, and the logic circuit isoperated at substantially maximum rated logic circuit clock frequencyand at a substantially maximum rated logic circuit operating voltage.(25) The method in embodiment (1), wherein one of the first selectedoperating mode and the second selected operating mode comprises a mode(Mode 2′) in which the processing unit is operated at less than maximumrated processing unit clock frequency and at less than a maximum ratedprocessing unit core voltage, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency and at asubstantially maximum rated logic circuit operating voltage. (26) Themethod in embodiment (1), wherein one of the first selected operatingmode and the second selected operating mode comprises a mode (Mode 2″)in which the processing unit is operated at less than maximum ratedprocessing unit clock frequency and at less than a maximum ratedprocessing unit core voltage, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency and at asubstantially maximum rated logic circuit operating voltage. (27) Themethod in embodiment (1), wherein one of the first selected operatingmode and the second selected operating mode comprises a mode (Mode 2″′)in which the processing unit is operated at less than maximum ratedprocessing unit clock frequency and at less than a maximum ratedprocessing unit core voltage just sufficient to maintain switchingcircuits in the processor unit at the processing unit clock frequency,and the logic circuit is operated at substantially maximum rated logiccircuit clock frequency and at a substantially maximum rated logiccircuit operating voltage. (28) The method in embodiment (1), whereinone of the first selected operating mode and the second selectedoperating mode comprises a mode (Mode 3) in which the processing unit isoperated at a slow but non-zero frequency processing unit clockfrequency and at less than or equal to a maximum rated processing unitcore voltage sufficient to maintain processor unit state, and the logiccircuit is operated at substantially maximum rated logic circuit clockfrequency and at a substantially maximum rated logic circuit operatingvoltage. (29) The method in embodiment (1), wherein one of the firstselected operating mode and the second selected operating mode comprisesa mode (Mode 3′) in which the processing unit is operated at asubstantially zero frequency processing unit clock frequency (clockstopped) and at less than or equal to a maximum rated processing unitcore voltage, and the logic circuit is operated at substantially maximumrated logic circuit clock frequency and at a substantially maximum ratedlogic circuit operating voltage. (30) The method in embodiment (1),wherein one of the first selected operating mode and the second selectedoperating mode comprises a mode (Mode 3″) in which the processing unitis operated at a substantially zero frequency processing unit clockfrequency (processing unit clock stopped) and at a processing unit corevoltage just sufficient to maintain processor unit state, and the logiccircuit is operated at substantially maximum rated logic circuit clockfrequency and at a substantially maximum rated logic circuit operatingvoltage. (31) The method in embodiment (1), wherein one of the firstselected operating mode and the second selected operating mode comprisesa mode (Mode 3″′) in which the processing unit is operated at asubstantially zero frequency processing unit clock frequency (processingunit clock stopped) and at a processing unit core voltage justsufficient to maintain processor unit state, and the logic circuit isoperated at a logic circuit clock frequency less than a maximum ratedlogic circuit clock frequency and at a logic circuit operating voltagethat is less than or equal to a maximum rated logic circuit operatingvoltage. (32) The method in embodiment (1), wherein one of the firstselected operating mode and the second selected operating mode comprisesa mode (Mode 3″″) in which the processing unit is operated at asubstantially zero frequency processing unit clock frequency (processingunit clock stopped) and at a processing unit core voltage justsufficient to maintain processor unit state, and the logic circuit isoperated at a logic circuit clock frequency less than a maximum ratedlogic circuit clock frequency and at a logic circuit operating voltagethat is less than a maximum rated logic circuit operating voltage. (33)The method in embodiment (1), wherein one of the first selectedoperating mode and the second selected operating mode comprises a mode(Mode 3″″′) in which the processing unit is operated at a substantiallyzero frequency processing unit clock frequency (processing unit clockstopped) and at a processing unit core voltage just sufficient tomaintain processor unit state, and the logic circuit is operated at asubstantially zero logic circuit clock frequency and at a logic circuitoperating voltage that is just sufficient to maintain logic circuitoperating state. (34) The method in embodiment (1), wherein one of thefirst selected operating mode and the second selected operating modecomprises a mode (Mode 4) in which the processing unit is powered off byremoving a processing unit clock frequency (processing unit clockstopped) and a processing unit core voltage. (35) The method inembodiment (1), wherein one of the first selected operating mode and thesecond selected operating mode comprises a mode (Mode 4′) in which theprocessing unit is powered off by removing a processing unit clockfrequency (processing unit clock stopped) and a processing unit corevoltage; and the logic circuit is powered off by removing the logiccircuit clock and by removing the logic circuit operating voltage or bysetting the logic circuit operating voltage below a level that willmaintain state, except that a real-time clock and circuit for waking thelogic circuit and the processing unit are maintained in operation. (36)The method in embodiment (1), wherein one of the first selectedoperating mode and the second selected operating mode comprises a mode(Mode 4″) in which the processing unit is powered off by removing aprocessing unit clock frequency (processing unit clock stopped) and aprocessing unit core voltage; and the logic circuit is powered off byremoving the logic circuit clock and by removing the logic circuitoperating voltage or by setting the logic circuit operating voltagebelow a level that will maintain state, except that a circuit for wakingthe logic circuit and the processing unit are maintained in operation.

(37) The method in embodiment (1), further comprising: while operatingin the second selected operating mode exhibiting that second selectedmode's characteristic power consumption range, (i) monitoring thecomputer system to detect the occurrence or non-occurrence of a secondevent; and (ii) transitioning the computer system from the secondselected operating mode to a third selected operating mode exhibitingthat third selected operating mode's power consumption range.

(38) The method in embodiment (1), wherein the first selected operatingmode and the second selected operating mode comprises differentoperating modes selected from the set of operating modes consisting of:(i) a mode in which the processing unit is operated at substantiallymaximum rated processing unit clock frequency and at substantiallymaximum rated processing unit core voltage, and the logic circuit isoperated at substantially maximum rated logic circuit clock frequency;(ii) a mode in which the processing unit is operated at less thanmaximum rated processing unit clock frequency and at less than or equalto a maximum rated processing unit core voltage, and the logic circuitis operated at substantially maximum rated logic circuit clockfrequency; and (iii) a mode in which the processing unit is operated ata substantially zero frequency processing unit clock frequency (clockstopped) and at less than or equal to a maximum rated processing unitcore voltage sufficient to maintain processor unit state, and the logiccircuit is operated at substantially maximum rated logic circuit clockfrequency.

(39) The method in embodiment (38), wherein the set further consists ofa mode in which the processing unit is powered off by removing aprocessing unit clock frequency (processing unit clock stopped) and aprocessing unit core voltage.

(40) The method in embodiment (1), further comprising: while operatingin the second selected operating mode exhibiting that second selectedmode's characteristic power consumption range, (i) monitoring thecomputer system to detect the occurrence or non-occurrence of a secondevent; and (ii) transitioning the computer system from the secondselected operating mode to a third selected operating mode exhibitingthat third selected operating mode's power consumption range.

(41) The method in embodiment (40), wherein the first selected operatingmode and the second selected operating mode comprises differentoperating modes, and the second selected operating mode and the thirdselected operating mode comprise different operating modes, each of thefirst, second, and third operating modes being selected from the set ofmodes consisting of: (i) a mode in which the processing unit is operatedat substantially maximum rated processing unit clock frequency and atsubstantially maximum rated processing unit core voltage, and the logiccircuit is operated at substantially maximum rated logic circuit clockfrequency; (ii) a mode in which the processing unit is operated at lessthan maximum rated processing unit clock frequency and at less than orequal to a maximum rated processing unit core voltage, and the logiccircuit is operated at substantially maximum rated logic circuit clockfrequency; and (iii) a mode in which the processing unit is operated ata substantially zero frequency processing unit clock frequency (clockstopped) and at less than or equal to a maximum rated processing unitcore voltage sufficient to maintain processor unit state, and the logiccircuit is operated at substantially maximum rated logic circuit clockfrequency.

(42) The method in embodiment (41), wherein the set further consists ofa mode in which the processing unit is powered off by removing aprocessing unit clock frequency (processing unit clock stopped) and aprocessing unit core voltage. (43) A computer program product for use inconjunction with a computer system including at least one processingunit, a memory coupled to the at least one processing unit, and logiccircuits coupled to the processing unit contributing to operation of thecomputer system, a method for controlling the operating mode and as aresult the power consumption of the computer system between a pluralityof operating modes each having a different electrical power consumptionlevels or ranges; the computer program product comprising a computerreadable storage medium and a computer program mechanism embeddedtherein, the computer program mechanism, comprising: a program modulethat directs the computer system to function in a specified manner, theprogram module including instructions for: (i) monitoring the computersystem to detect the occurrence or non-occurrence of a first event whileoperating in a first selected operating mode exhibiting that firstselected mode's characteristic power consumption range; and (ii)transitioning the computer system from the first selected operating modeto a second selected operating mode exhibiting that second selectedoperating mode's power consumption range. (44) The computer programproduct in embodiment (43), wherein the program module further includinginstructions for: while operating in the second selected operating modeexhibiting that second selected mode's characteristic power consumptionrange, (i) monitoring the computer system to detect the occurrence ornon-occurrence of a second event; and (ii) transitioning the computersystem from the second selected operating mode to a third selectedoperating mode exhibiting that third selected operating mode's powerconsumption range. (45) The computer program product in embodiment (44),wherein the first selected operating mode and the second selectedoperating mode comprises different operating modes, and the secondselected operating mode and the third selected operating mode comprisedifferent operating modes, each of the first, second, and thirdoperating modes being selected from the set of modes consisting of: (i)a mode in which the processing unit is operated at substantially maximumrated processing unit clock frequency and at substantially maximum ratedprocessing unit core voltage, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency; (ii) a modein which the processing unit is operated at less than maximum ratedprocessing unit clock frequency and at less than or equal to a maximumrated processing unit core voltage, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency; and (iii) amode in which the processing unit is operated at a substantially zerofrequency processing unit clock frequency and at less than or equal to amaximum rated processing unit core voltage sufficient to maintainprocessor unit state, and the logic circuit is operated at substantiallymaximum rated logic circuit clock frequency. (46) The computer programproduct in embodiment (45), wherein the set further consists of a modein which the processing unit is powered off by removing a processingunit clock frequency and a processing unit core voltage.

(47) A computer system comprising: at least one processing unit and amemory coupled to the at least one processing unit; and logic circuitscoupled to the processing unit contributing to operation of the computersystem; a controller for controlling the operating mode and as a result,the power consumption of the computer system between a plurality ofoperating modes each having a different electrical power consumptionlevels or ranges; the controller being operable while operating in afirst selected operating mode exhibiting that first selected mode'scharacteristic power consumption range, (i) to monitor the computersystem to detect the occurrence or non-occurrence of a first event; and(ii) to transition the computer system from the first selected operatingmode to a second selected operating mode exhibiting that second selectedoperating mode's power consumption range.

In a fourth group of innovations, the invention provides variousembodiments associated with Apparatus, Architecture, and Method forIntegrated Modular Server System Providing Dynamically Power-managed andWork-load Managed Network Devices.

(1) An integrated server system unit comprising: a frame, housing, orenclosure; a plurality of network devices disposed within the frame,housing, or enclosure; the network devices including at least onenetwork device operating as a computer node, at least one network deviceoperating as a monitor node, and at least one network device operatingas a switching node; and each the computer node, the monitor node, andthe switching node being coupled for communication.

(2) A system as in embodiment (1), wherein at least two of the computernode, the monitor node, and the switching node are provided by a commonphysical device. (3) A system as in embodiment (1), wherein theintegrated server system includes a plurality of computer nodes. (4) Asystem as in embodiment (1), wherein the computer node comprises aserver module. (5) A system as in embodiment (1), wherein the monitornode comprises a management module. (6) A system as in embodiment (1),wherein the switching node comprises a switch module. (7) A system as inembodiment (1), wherein: the computer node comprises a server module;the monitor node comprises a management module; and switching nodecomprises a switch module. (8) A system as in embodiment (7), whereinthe computer node is power managed to adjust performance to requiredlevel and to consume only so much electrical power or energy as tosatisfy the required performance level. (9) A system as in embodiment(7), wherein each of the computer node, the management node, and theswitch node are power managed. (10) A system as in embodiment (8),wherein the computer node is power managed by adjusting processorperformance to at least one of (i) a predicted processor processingrequirement and (ii) a measured processor processing requirement. (11) Asystem as in embodiment (10), wherein the predicted processor processingrequirement when used is a Quality of Service (QoS) based requirement,and the measured processor processing requirement when used comprises asubstantially real-time measured processor processing requirement. (12)A system as in embodiment (11), wherein the substantially real-timeprocessor processing requirement comprises an idle thread executiondetection and response thereto. (13) A system as in embodiment (9),wherein the management node is power managed by adjusting a processoroperation within the management node to provide execution ofinstructions that permit the management node to sample system loadingand adjust other nodes within the system frequently enough to provide adesired power management adjustment update frequency for each node beingpower managed. (14) A system as in embodiment (13), wherein theprocessor operation is adjusted by adjusting a management node processorclock frequency and/or a management node processor core voltage. (15) Asystem as in embodiment (13), wherein the computer node network devicesmay be configured or reconfigured as any or all of a web server, astreaming media server, a cache server, an application server, a fileserver, a router, and combinations thereof. (16) A system as inembodiment (15), wherein the configuration may be performed dynamicallyduring operation of the computer node network device during operation ofthe system so that the system has a number of computer node networkdevices sufficient to satisfy a current demand on the network fornetwork devices of that type. (17) A system as in embodiment (16),wherein the configuration or reconfiguration may controlled by a monitornode or a management module, or by a self-reconfiguration procedurewithin the computer node network device.

(18) An dynamically adaptable integrated server system unit comprising:a frame, housing, or enclosure; a plurality of configurable anddynamically reconfigurable network devices disposed within the frame,housing, or enclosure; and the dynamically reconfigurable networkdevices including at least one network device operating as a computernode.

(19) A system as in embodiment (18), wherein the computer node comprisesa server module. (20) A system as in embodiment (19), wherein the servermodule is initially configurable and subsequently reconfigurable duringoperation as any one or more or all of a web server, a streaming mediaserver, a cache server, a file server, an application server, a router,and combinations thereof. (21) A system as in embodiment (18), whereineach computer node is configurable and reconfigurable as a combinationtype of network device, and wherein the network device configured in thecomputer node is a network device selected from the set consisting of aweb server, a streaming media server, a cache server, a file server, anapplication server, a router, and combinations thereof. (22) A system asin embodiment (18), wherein the network device is dynamically andadaptably reconfigurable at any time based on types of activitiesdetected within a network to which the network device is then presentlyor will be connected. (23) A system as in embodiment (22), wherein whenthe network device is configured prospectively for a network that thenetwork device is to be connected, the activities are detected by eitherthe network device in a different configuration or by a differentnetwork device coupled to the network. (24) A system as in embodiment(22), wherein the system includes a network activity monitor thatdetects the types of activities present on a network to which theactivity monitor is coupled. (25) A system as in embodiment (24),wherein the types of activities present on a network to which theactivity monitor is coupled monitored by the activity monitor include atleast one of a number or volume of web pages served, a number or volumeof streaming media served, a number or volume of files served, a numberor volume of applications served, a number or volume of cached dataserved, an amount of network traffic routed, and combinations thereof.(26) A system as in embodiment (25), wherein the activity monitor isbased in part on detection of the execution of idle threads by aprocessor within the network device during periods of time when thenetwork device is not performing the activity for which the networkdevice is configured. (27) A system as in embodiment (18), wherein theserver module may be reconfigured at any time based on any one or moreof server processor loading, quality of service indicator, orcombinations thereof. (28) A system as in embodiment (19), wherein theserver module may be reconfigured at any time based on any one or moreof server processor loading, quality of service indicator, orcombinations thereof. (29) A system as in embodiment (18), wherein thereconfiguration of a computer node is initiated by any computer nodetype network device including the same computer node as is beingreconfigured. (30) A system as in embodiment (18), wherein thereconfiguration of a computer node is initiated by any computer nodethat has been designated as a master computer node. (31) A system as inembodiment (30), wherein any computer node may be designated as themaster node. (32) A system as in embodiment (31), wherein a particularcomputer node is designated as a master node on the basis of itsposition within a chassis. (33) A system as in embodiment (31), whereina particular computer node is designated as a master node on the basisof the order of power-up or boot completion. (34) A system as inembodiment (18), wherein reconfiguration of the computer node comprisesaltering the software and/or firmware instructing the computer node.(35) A system as in embodiment (22), wherein reconfiguration of thecomputer node comprises altering the software and/or firmwareinstructing the computer node. (36) A system as in embodiment (18),wherein reconfiguration of the computer node comprises altering the dataorganization of a data storage device integral with or coupled to thecomputer node. (37) A system as in embodiment (22), whereinreconfiguration of the computer node comprises altering the dataorganization of a data storage device integral with or coupled to thecomputer node. (38) A system as in embodiment (37), wherein the datastorage device comprises a hard disc drive based RAID storage array andaltering the data organization comprises altering a RAID configurationof the data to provide better performance for the type of data beingserved. (39) A system as in embodiment (18), wherein the reconfigurationof a computer node is initiated by a management node type networkdevice. (40) A system as in embodiment (18), wherein the dynamicallyreconfigurable network devices include a plurality of network devicesconfigured as the same type. (41) A system as in embodiment (40),wherein a plurality of network devices of the same type are groupedtogether into a group that are treated as a single network device. (42)A system as in embodiment (41), wherein the group of network devicestreated as a single network device is managed and controlled as a singlenetwork device. (43) A system as in embodiment (42), wherein the groupof network devices treated as a single network device is power managedas a single network device. (44) A system as in embodiment (42), whereinthe group of network devices treated as a single network device ismonitored as a single network device. (45) A system as in embodiment(41), wherein the plurality of grouped network devices are coupled forcommunication and the logical grouping of the plurality of networkdevices into a single logical network device is performed under controlof software or firmware. (46) A system as in embodiment (41), whereinthe plurality of grouped network devices are electrically coupled via abackplane bus and the logical grouping of the plurality of networkdevices into a single logical network device is performed under controlof software. (47) A system as in embodiment (46), wherein the softwareexecutes within a processor and memory associated with each networkdevice. (48) A system as in embodiment (44), wherein the plurality ofnetwork devices each comprise a server group. (49) A system as inembodiment (44), wherein the plurality of network devices each comprisea computer node. (50) A system as in embodiment (49), wherein eachcomputer node is configured as a computer node selected from the groupconsisting of a web server, a streaming media server, a cache server, afile server, an application server, and a router.

In a fifth group of innovations, the invention provides variousembodiments associated with System, Architecture, and Method for LogicalServer and Other Network Devices in a Dynamically ConfigurableMulti-server Network Environment.

(1) In a system having a plurality of physical network devices, a methodfor grouping the plurality of physical network devices into a logicalnetwork device for operation and control, the method comprising:coupling the physical network devices that are to be grouped together asa logical device for communication; identifying a manager to superviseoperation of the logical device; receiving a request for performance ofa task; and executing a procedure within the manager to coordinate andcontrol operation of the logical device to perform the task, thecoordination and control including: (i) determining a capability of eachthe physical network device, and (ii) distributing network tasks betweenand among the network devices based on the determined capabilities andrequests submitted to the network.

(2) The method in embodiment (1), wherein the manager supervisingoperation is performed at least in part by a component of one of theplurality of network devices. (3) The method in embodiment (1), whereinthe manager supervising operation is performed at least in part by acomponent of one of the plurality of network devices comprising thelogical network device. (4) The method in embodiment (1), wherein themanager supervising operation is performed at least in part by anexternal manager separate from the plurality of network devicescomprising the logical network device. (5) The method in embodiment (1),wherein the manager supervising operation is performed by a managementmodule and the plurality of physical network devices comprise servermodules. (6) The method in embodiment (1), wherein the managersupervising operation is performed by a server module designated as amaster. (7) The method in embodiment (6), wherein the plurality ofnetwork devices comprise server modules, and the manager supervisingoperation is performed by one of the server modules designated as amaster. (8) The method in embodiment (6), wherein the plurality ofnetwork devices comprise server modules, and the manager supervisingoperation is performed by a server module different from the pluralityof server modules grouped as a logical network device and designated asa master. (9) The method in embodiment (1), wherein the plurality oflogical devices comprise switch modules. (10) The method in embodiment(1), wherein the plurality of logical devices comprise managementmodules. (11) The method in embodiment (1), wherein the plurality oflogical devices comprise network devices selected from the setconsisting of management modules, server modules, and switching modules.(12) The method in embodiment (1), wherein the plurality of logicaldevices are configured as a logical manager. (13) The method inembodiment (1), wherein the plurality of logical devices are configuredas a logical switcher. (14) The method in embodiment (1), wherein theplurality of logical devices are configured as a logical router. (15)The method in embodiment (1), wherein the plurality of logical devicesare configured as a logical server. ( 16) The method in embodiment (1),wherein the plurality of logical devices comprise server modules. (17)The method in embodiment (1), wherein the plurality of logical devicescomprise server modules and the logical network device comprises alogical server module configured as one of a web server, a streamingmedia server, a cache server, a file server, an application server, andcombinations thereof. (18) The method in embodiment (1), wherein theplurality of physical devices include a plurality of data storagedevices, and the coordination and control of the physical devices intothe logical device comprise managing the storage devices as a redundantarray of independent disks (RAID). (19) The method in embodiment (18),wherein the plurality of data storage devices comprise a plurality ofdisk drives managed as a mirroring data storage subsystem. (20) Themethod in embodiment (18), wherein the RAID is managed as either a RAIDLevel 1 or RAID Level 0+1. (21) The method in embodiment (18), whereinthe plurality of storage devices comprise hard disk drives and operationof the hard disk drives is managed to reduce power consumption andprolong disk drive life. (22) The method in embodiment (19), wherein themanaging includes: selecting a first set of disc drives from among aplurality of sets of disc drives as a currently active set; selecting asecond set of disc drives as a currently inactive set and placing theselected inactive set in a power conserving mode; using the first set ofdisc drives to retrieve data in response to read data requests;activating the second set of disc drives in response to a write datarequest prior to performing the requested write operation; performingthe write operation to both the first and second set of disc drives; andselecting one of the first set and second set as the inactive set anddeactivating the selected inactive set after performing the writeoperation. (23) The method in embodiment (22), wherein the disc drivesare configured as a RAID Level 1. (24) The method in embodiment (22),wherein the disc drives are configured as a RAID Level 0+1. (25) Themethod in embodiment (22), wherein the disc drives are configured asmirrored storage each set duplicating the data of the other set. (26)The method in embodiment (1), wherein the plurality of grouped networkdevices are electrically coupled via a backplane bus and the logicalgrouping of the plurality of network devices into a single logicalnetwork device is performed under control of software. (27) The methodin embodiment (1), wherein the software executes within a processor andmemory associated within each network device. (28) The method inembodiment (1), wherein a plurality of network devices of the same typeare grouped together and treated as a single network device. (29) Themethod in embodiment (1), wherein the group of network devices treatedas a single network device is managed and controlled as a single networkdevice. (30) The method in embodiment (1), wherein the group of networkdevices treated as a single network device is power managed as a singlenetwork device. (31) The method in embodiment (1), wherein the group ofnetwork devices treated as a single network device is monitored as asingle network device. (32) The method in embodiment (1), wherein theplurality of grouped network devices are electrically coupled via abackplane bus and the logical grouping of the plurality of networkdevices into a single logical network device is performed under controlof software. (33) The method in embodiment (1), wherein the softwareexecutes within a processor and memory associated within each networkdevice. (34) The method in embodiment (1), wherein the plurality ofnetwork devices each comprise a server group. (35) The method inembodiment (34), wherein each the server group comprises a plurality ofserver modules. (36) The method in embodiment (1), wherein the pluralityof network devices each comprise a computer node. (37) The method inembodiment (36), wherein each computer node is configured as a computernode selected from the group consisting of a web server, a streamingmedia server, a cache server, a file server, an application server, anda router. (38) The method in embodiment (1), wherein the logical deviceis managed as a single image. (39) The method in embodiment (1), whereinthe method includes a first plurality of physical devices grouped as afirst logical device, and a second plurality of physical devices groupedas a second logical devices. (40) The method in embodiment (39), whereinthe first and second logical devices are power managed as logicaldevices. (41) The method in embodiment (1), wherein tasks performed byeach of the logical devices are monitored for each logical device, foreach physical device within the logical device, or both. (42) The methodin embodiment (1), wherein tasks performed by each of the logicaldevices are monitored for each logical device, for each physical devicewithin the logical device, or both; and the task monitoring informationis used to power manage the logical device and the physical devices.(43) The method in embodiment (42), wherein the power managementincludes reconfiguring the logical unit to provide a requiredperformance level with a desired power consumption. (44) The method inembodiment (42), wherein the power management includes reconfiguring thelogical unit to minimize power consumption while maintaining a requiredtask performance level. (45) The method in embodiment (44), wherein thereconfiguring the logical unit to minimize power consumption whilemaintaining a required task performance level is performed in accordancewith power management policies. (46) The method in embodiment (44),wherein the reconfiguring the logical unit to extend the life of acomponent constituting the physical device. (47) The method inembodiment (45), wherein the power management policies include a policythat attempts to satisfy performance requirements by operating each offirst selected physical devices at first performance levels, and a eachof second selected physical devices at second performance levels. (48)The method in embodiment (47), wherein the second performance levels area performance level at which substantially no tasks are performed andpower consumption is reduced relative to the first performance levels.(49) The method in embodiment (48), wherein the first performance levelsare between about 40 percent and about 100 percent of maximumperformance level. (50) The method in embodiment (45), wherein the powermanagement policies include a policy that attempts to satisfyperformance requirements by operating a minimum number of physicaldevices to satisfy task performance requirements. (51) The method inembodiment (40), wherein performance of a logical group of physicalnetwork devices forming a single logical device are power managed byreducing the performance and power consumption of each constituentphysical device in predetermined equal increments, predetermined unequalincrements, in a substantially continuous manner, or dynamically in acontinuous or incremental manner. (52) The method in embodiment (51),wherein the unequal increments include placing one or more of theplurality of physical devices in an operating mode wherein the physicaldevice includes a processor operating to perform tasks only when aprocessor clock signal is provided, and the processor clock signal isstopped or substantially stopped. (53) The method in embodiment (52),wherein the processor is placed in a suspend mode. (54) The method inembodiment (51), wherein the unequal increments include placing one ormore of the plurality of physical devices in a sleep mode. (55) Themethod in embodiment (51), wherein the unequal increments includeplacing one or more of the plurality of physical devices in apowered-off mode. (55) The method in embodiment (45), wherein acomposite performance of a logical network device is achieved by placingsome physical network devices in a first power saving mode and byplacing others in a different mode.

(51) A method of reducing power consumption in a computer system havinga mirroring data storage subsystem, the method comprising steps of:selecting a first set of disc drives from among a plurality of sets ofdisc drives as a currently active set; selecting a second set of discdrives as a currently inactive set and placing the selected inactive setin a power conserving mode; using the first set of disc drives toretrieve data in response to read data requests; activating the secondset of disc drives in response to a write data request prior toperforming the requested write operation; performing the write operationto both the first and second set of disc drives; and selecting one ofthe first set and second set as the inactive set and deactivating theselected inactive set after performing the write operation.

(52) The method in embodiment (51), wherein the disc drives areconfigured as a RAID Level 1. (53) The method in embodiment (51),wherein the disc drives are configured as a RAID Level 0+1. (54) Themethod in embodiment (51), wherein the disc drives are configured asmirrored storage each set duplicating the data of the other set. (55)The method in embodiment (51), wherein: the disc drives are disposed onseparate server modules that have been grouped as a single logicaldevice; and the disk drives are configured as either a RAID Level 1 or aRAID Level 0+1 and include data mirroring.

In a sixth group of innovations, the invention provides variousembodiments associated with Apparatus and Method for Modular DynamicallyPower-Managed Power Supply and Cooling System for Computer Systems,Server Applications, and Other Electronic Devices.

(1) An electrical apparatus comprising: a frame or enclosure; at leastone electrical circuit drawing electrical power in the form of analternating or direct electrical voltage, current, or a combination ofan electrical voltage and an electrical current disposed within theframe or enclosure, the electrical circuit utilizing the electricalpower and generating heat as a result of the utilization; at least onetemperature sensor within the enclosure for monitoring and reporting thetemperature proximate the sensor to a temperature monitor; and a powermanager receiving the reported temperature and controlling thetemperature at the temperature sensor by controlling electrical powerdrawn by the electrical circuit and thereby the heat generated byoperation of the circuit.

(2) The apparatus in embodiment (1), wherein the at least one electricalcircuit comprises a computer having a processor receiving an operatingvoltage and a processor clock signal. (3) The apparatus in embodiment(2), wherein the computer is configured as a server. (4) The apparatusin embodiment (3), wherein the power manager comprises a powermanagement circuit. (5) The apparatus in embodiment (3), wherein theserver comprises a server module and the power manager comprises amanagement module. (6) The apparatus in embodiment (1), wherein theapparatus comprises a plurality of the electrical circuits eachincluding a computer having a processor receiving an operating voltageand a processor clock signal. (7) The apparatus in embodiment (6),wherein the power manager controls the electrical power drawn and theheat generated by the electrical circuits by controlling either thefrequency of the processor clock signal, or the operating voltage, or acombination of the processor clock frequency and the processor operatingvoltage. (8) The apparatus in embodiment (7), wherein the power managerreduces the electrical power drawn by the electrical circuits bymonitoring the temperature sensor and controlling an output signalgenerated at least in part by the temperature sensor to be within apredetermined range. (9) The apparatus in embodiment (8), wherein thepredetermined range includes a predetermined maximum. (10) The apparatusin embodiment (6), wherein at least some of the plurality of electricalcircuits are configured as network devices including the processorreceiving the operating voltage and the processor clock signal; and thepower manager controls the electrical power drawn and the heat generatedby the network devices by controlling either the frequency of theprocessor clock signal, or the operating voltage, or a combination ofthe processor clock frequency and the processor operating voltage. (11)The apparatus in embodiment (10), wherein at least some of the networkdevices comprise circuits configured as a network device selected fromthe set consisting of a web server, a streaming media server, a cacheserver, a file server, an application server, and a router. (12) Theapparatus in embodiment (10), wherein at least some of the networkdevices comprise server computers that further include at least one harddisk drive for storing data or other content to be served and a networkcommunication circuit for communicating with an external client over acommunication link. (13) The apparatus in embodiment (10), wherein theserver computers comprises server modules and the power managercomprises at least one management module. (14) The apparatus inembodiment (10), wherein the configured network device comprises amanagement node type network device. (15) The apparatus in embodiment(10), wherein the system includes a plurality of temperature sensorswithin the enclosure reporting to one or more network devices. (16) Theapparatus in embodiment (15), wherein the plurality of temperaturesensors are spatially distributed to provide temperature monitoring ofdifferent network devices within the enclosure. (17) The apparatus inembodiment (15), wherein the plurality of temperature sensors arespatially distributed to provide temperature monitoring of differentnetwork devices and power supplies within the enclosure. (18) Theapparatus in embodiment (12), wherein when the temperature sensed by atemperature sensor is within a predetermined magnitude relationship of afirst predetermined value at least one network device is transitioned toa lower power consumption state thereby generating less heat. (19) Theapparatus in embodiment (18), wherein when the temperature sensed by atemperature sensor is within a predetermined magnitude relationship of asecond predetermined value at least one network device is transitionedto a powered off state. (20) The apparatus in embodiment (1), whereinthe operational state of at least one network device is reduced to alower power consuming and heat dissipating state in response to atemperature sensor reporting a temperature greater than or equal to apredetermined value. (21) The apparatus in embodiment (20), whereinafter the power consumption state has been lowered permitting thenetwork device to be operated at a higher power consuming state when thetemperature sensed is below a predetermined temperature value, the lowertemperature value being selected to provide hysteresis and preventoscillation between higher power state and lower powered state. (22) Theapparatus in embodiment (1), when the temperature sensed by atemperature sensor is within a predetermined magnitude relationship of afirst predetermined value at least one network device is transitioned toa lower power consumption state. (23) The apparatus in embodiment (22),wherein the lower power consumption state is achieved by lowering theclock frequency of the processor, the clock frequency of a bus couplinga processor to other components, or the operating voltage of theprocessor or other components. (24) The apparatus in embodiment (22),wherein additional networked devices are sent to lower energy consumingmodes if the temperature remains above a predetermined temperaturevalue. (25) The apparatus in embodiment (7), wherein the controlling ofeither the frequency of the processor clock signal, or the operatingvoltage, or a combination of the processor clock frequency and theprocessor operating voltage, is controlled by a computer programexecuting instructions to implement a control procedure at least in partin at least one of the processors of the computers that transition oneor more of the processors between different operating modes havingdifferent electrical power consumptions and different heat generation;the procedure including: while operating in a first selected operatingmode exhibiting that first selected mode's characteristic powerconsumption range, (i) monitoring the computer system to detect theoccurrence or non-occurrence of a first event; and (ii) transitioningthe computer system from the first selected operating mode to a secondselected operating mode exhibiting that second selected operating mode'spower consumption range.

(26) The apparatus in embodiment (25), wherein the procedure furtherincluding: while operating in the second selected operating modeexhibiting that second selected mode's characteristic power consumptionrange, (i) monitoring the computer system to detect the occurrence ornon-occurrence of a second event; and (ii) transitioning the computersystem from the second selected operating mode to a third selectedoperating mode exhibiting that third selected operating mode's powerconsumption range.

(27) The apparatus in embodiment (26), wherein the first selectedoperating mode and the second selected operating mode comprisesdifferent operating modes, and the second selected operating mode andthe third selected operating mode comprise different operating modes,each of the first, second, and third operating modes being selected fromthe set of modes consisting of: (i) a mode in which the processing unitis operated at substantially maximum rated processing unit clockfrequency and at substantially maximum rated processing unit corevoltage, and the logic circuit is operated at substantially maximumrated logic circuit clock frequency; (ii) a mode in which the processingunit is operated at less than maximum rated processing unit clockfrequency and at less than or equal to a maximum rated processing unitcore voltage, and the logic circuit is operated at substantiallymaximum, rated logic circuit clock frequency; and (iii) a mode in whichthe processing unit is operated at a substantially zero frequencyprocessing unit clock frequency (clock stopped) and at less than orequal to a maximum rated processing unit core voltage sufficient tomaintain processor unit state, and the logic circuit is operated atsubstantially maximum rated logic circuit clock frequency. (28) Theapparatus in embodiment (27), wherein the set further consists of a modein which the processing unit is powered off by removing a processingunit clock frequency (processing unit clock stopped) and a processingunit core voltage. (29) The apparatus in embodiment (1), furthercomprising at least one cooling fan and the apparatus controlling aspeed of the fan, including an on/off condition of the fan, to achieve adesired temperature at the sensor. (30) The apparatus in embodiment(29), wherein the fan is not rotated and passive cooling is used whenelectrical power drawn and heat generated are sufficiently small topermit such passive cooling while maintaining a predeterminedtemperature range. (31) The apparatus in embodiment (29), wherein theapparatus includes a plurality of cooling fans and the plurality ofcooling fans are controlled to achieve a desired temperature. (32) Theapparatus in embodiment (31), wherein the apparatus further includes aplurality of temperature sensors and the plurality of cooling fans areoperated in a coordinated manner to achieve a desired temperature rangeproximate at least some of the temperature sensors. (33) The apparatusin embodiment (31), wherein the cooling fans are modular cooling fanunits that provide mechanical connectors and electrical circuits toprovide powered-on hot-swappability. (34) The apparatus in embodiment(33), wherein the modular cooling fan units are organized into coolingfan banks that provide mechanical connectors and electrical circuits toprovide powered-on hot-swappability. (35) The apparatus in embodiment(34), wherein the at least two banks of three cooling fan units areprovided at different locations within the frame or enclosure. (36) Theapparatus in embodiment (33), wherein the cooling fan units includefail-over protection circuits. (37) The apparatus in embodiment (31),wherein different ones of the plurality of cooling fan units areoperated or not operated in a coordinated manner to provide desiredcooling of the apparatus and to achieve a desired life cycle and/orreliability for the cooling fans. (38) The apparatus in embodiment (31),wherein different ones of the plurality of cooling fan units areoperated or not operated or operated at different speeds in acoordinated manner to provide desired cooling of the apparatus and toprovide such cooling at a minimum aggregate cooling fan powerconsumption. (39) The apparatus in embodiment (1), wherein powerconsumption within the apparatus is further reduced by adjusting thenumber and motor speed of cooling fans responsible for cooling theapparatus. (40) The apparatus in embodiment (11), wherein the apparatusfurther includes a plurality of temperature sensors and a plurality ofcooling devices, the cooling devices operating under control of acontrol device that controls each cooling device to provide cooling atthe rate and location desired to maintain the network devices within apredetermined operating temperature range. (41) The apparatus inembodiment (40), wherein a plurality of temperature sensors are disposedin the frame of enclosure and a plurality of cooling devices aredisposed within the enclosure, the plurality of temperature sensorscommunicating a temperature signal to a control means and the controlmeans adjusting the on/off status and operational parameters of thecooling units to extract heat according to predetermined rules. (42) Theapparatus in embodiment (41), wherein the cooling devices comprise motordriven fans. (43) The apparatus in embodiment (41), wherein the coolingdevices comprise valves controlling the circulation of a cooling fluid.(44) The apparatus in embodiment (41), wherein the cooling devicescomprise conductive heat exchangers. (45) The apparatus in embodiment(41), wherein the cooling devices comprise convective heat exchangers.(46) The apparatus in embodiment (10), wherein: the server computerscomprises server modules and the power manager comprises at least onemanagement module; power consumption within the apparatus is controlledreduced by adjusting the number and motor speed of cooling fansresponsible for cooling the apparatus. (47) The apparatus in embodiment(11), wherein the apparatus further includes a plurality of temperaturesensors and a plurality of cooling devices, the cooling devicesoperating under control of a control device that controls each coolingdevice to provide cooling at the rate and location desired to maintainthe network devices within a predetermined operating temperature range.(48) The apparatus in embodiment (47), wherein a plurality oftemperature sensors are disposed in the frame of enclosure and aplurality of cooling devices are disposed within the enclosure, theplurality of temperature sensors communicating a temperature signal to acontrol means and the control means adjusting the on/off status andoperational parameters of the cooling units to extract heat according topredetermined rules. (49) A system as in embodiment (48), wherein therotational speed of a motor drive cooling is adjusted to maintain apredetermined temperature range proximate a temperature sensor. (50) Asystem as in embodiment (48), wherein the rotational speed of a motordrive cooling is adjusted to maintain a predetermined temperature rangewithin an enclosure. (51) A system as in embodiment (48), wherein theamount of heat extracted from an enclosure is adjusted to maintain apredetermined temperature and reduce power consumed by the coolingdevice. (52) A system as in embodiment (48), wherein the heat extractorcomprises a motor driven cooling device. (53) The apparatus inembodiment (1), further including a plurality of power supplies whereinthe plurality of power supplies are controlled to maintain a requiredpower output level drawn by the at least one electrical circuit and tooperate the power supplies according to predetermined power supplymanagement policy. (54) The apparatus in embodiment (53), whereinoperating the plurality of power supplies at a preferred efficiencyincludes operating at least some of the power supplies a preferredoutput and/or efficiency at a partial electrical output loading lessthan a maximum loading to extend a lifetime of the power supplies. (55)The apparatus in embodiment (53), wherein operating the plurality ofpower supplies according to the policy includes operating at least someof the power supplies at up to a maximum rating and not operating otherof the plurality of power supplies so that the aggregate power consumedby the apparatus including power lost in operation of the power suppliesis reduced. (56) The apparatus in embodiment (55), wherein the powersupplies comprise battery power supplies. (57) The apparatus inembodiment (55), wherein the power supplies comprise power suppliesreceiving an alternating current utility line voltage and current andgenerating at least one direct current voltage and current. (58) Theapparatus in embodiment (57), wherein the alternating current utilityline (ac) voltage is a voltage substantially in the range of betweenabout 90 volts and substantially 300 volts, and the direct current (dc)voltage is in the range of between about ±0.5 volt and about ±20 volts.(59) The apparatus in embodiment (57), wherein the alternating currentutility line (ac) voltage is a voltage substantially in the range ofbetween substantially 100 volts and 130 volts, and the direct current(dc) voltage is in the range of between about 1 volt and about 5 volts.(60) The apparatus in embodiment (57), wherein the power supplymanagement policy further includes automatically alternating a pluralityof power supplies so that the aggregate plurality of power supplies areoperated efficiently and have an extended lifetime. (61) The apparatusin embodiment (60), wherein the automatically alternating the pluralityof power supplies includes changing the electrical power that may bedrawn from each of the plurality of power supplies under computercontrol so that the aggregate plurality of power supplies are operatedefficiently and have an extended lifetime. (62) The apparatus inembodiment (53), wherein only selected ones of the plurality of powersupplies are operated. (63) The apparatus in embodiment (53), whereinmultiple ones of the power supplies are operated concurrently but eachis operated at less than rated power output capacity. (64) The apparatusin embodiment (53), wherein the plurality of power supply units includefail-over protection circuits. (65) The apparatus in embodiment (53),wherein the elapsed time and/or power supply loading history aremonitored and stored in a non-volatile memory store and used with thepower supply management policy. (66) The apparatus in embodiment (65),wherein the stored history are utilized to predict failure and/orequalize lifetime of the power supplies according to a power supplylifetime prediction routine. (67) The apparatus in embodiment (66),wherein the power supply lifetime prediction routine is statisticallybased prediction routine utilizing a lifetime and failure model adaptedto each particular type of power supply. (68) The apparatus inembodiment (53), wherein the plurality of power supplies comprise powersupplies having different output characteristics types and thecombination of power supplies providing electrical operating power tosatisfy electrical loading at any particular time and having a desiredaggregate operating characteristic are dynamically selected. (69) Theapparatus in embodiment (68), wherein the desired aggregate operatingcharacteristic is a substantially minimized power consumption at therequired power output.

(70) A power-conservative multi-node network device, comprising: anenclosure having a power supply and a back-plane bus; a plurality ofhot-pluggable node devices in the form of printed circuit (PC) cardsadapted for connection with the back-plane buss; and each the nodedevice being reconfigurable in substantially real-time to adapt tochanging conditions on the network.

(71) The power-conservative multi-node network device in embodiment(70), wherein the plurality of hot-pluggable node devices comprisesixteen node devices. (72) The power-conservative multi-node networkdevice in embodiment (70), wherein each of the node devices includespower saving control features.

Numerous other embodiments and features of the invention are describedin the specification and drawings even though not specificallyhighlighted here.

Aspects of the invention though often described in the context ofprocessors, CPUs, network devices, servers, and the like; haveparticular benefits relative to power and evergy conservation whenapplied to server farms where large quantities of energy are conserveddirectly as a result of lower power operation without performancesacrifice as well as energy conserved as a result of higher density andlower facilities space and cooling requirements.

Those workers having ordinary skill in the art in light of thedescription provided will no doubt appreciate other aspects, features,and advantages of the inventive system, method, and software control. Itwill be appreciated that the afore described procedures implemented in acomputer environment may be implemented using hardware, software, and/orfirmware, and combinations of these. The detection, analysis,monitoring, decision making, and control functions are particularlyamenable to computer program software and firmware implementations andmay readily be implemented in a central processing unit (CPU),processor, controller, micro-controller, or other logic unit within orassociated with the computers. Therefore the invention includes hardwareand software implementations, and descriptions of procedures and methodsanticipate that such procedures and methods may be implemented as acomputer program and computer program product.

The foregoing descriptions of specific embodiments of the presentinvention have been presented for purposes of illustration anddescription. They are not intended to be exhaustive or to limit theinvention to the precise forms disclosed, and obviously manymodifications and variations are possible in light of the aboveteaching. The embodiments were chosen and described in order to bestexplain the principles of the invention and its practical application,to thereby enable others skilled in the art to best utilize theinvention and various embodiments with various modifications as aresuited to the particular use contemplated. It is intended that the scopeof the invention be defined by the claims appended hereto and theirequivalents.

All publications, product or other data sheets, web-site content, andpatent applications cited or referenced in this specification are hereinincorporated by reference as if each individual publication or patentapplication were specifically and individually indicated to beincorporated by reference.

1. In a system having a plurality of physical network devices, a methodfor grouping said plurality of physical network devices into a logicalnetwork device for operation and control, said method comprising:coupling said physical network devices that are to be grouped togetheras a logical device for communication; identifying a manager tosupervise operation of said logical device; identifying a number of saidplurality of physical network devices that are configured to perform atask; receiving at least one request for performance of the task; anddetermining demand for the task; and based on the demand, executing aprocedure within said manager to coordinate and control operation ofsaid logical device to increase or decrease the number of physicalnetwork devices that are configured to perform the task.
 2. The methodin claim 1, wherein said manager supervising operation is performed atleast in part by a component of one of said plurality of networkdevices.
 3. The method in claim 1, wherein said manager supervisingoperation is performed at least in part by a component of one of saidplurality of network devices comprising said logical network device. 4.The method in claim 1, wherein said manager supervising operation isperformed at least in part by an external manager separate from saidplurality of network devices comprising said logical network device. 5.The method in claim 1, wherein said manager supervising operation isperformed by a management module and said plurality of physical networkdevices comprise server modules.
 6. The method in claim 1, wherein saidmanager supervising operation is performed by a server module designatedas a master.
 7. The method in claim 6, wherein said plurality of networkdevices comprise server modules, and said manager supervising operationis performed by one of said server modules designated as a master. 8.The method in claim 6, wherein said plurality of network devicescomprise server modules, and said manager supervising operation isperformed by a server module different from said plurality of servermodules grouped as a logical network device and designated as a master.9. The method in claim 1, wherein said plurality of logical devicescomprise switch modules.
 10. The method in claim 1, wherein saidplurality of logical devices comprise management modules.
 11. The methodin claim 1, wherein said plurality of logical devices comprise networkdevices selected from the set consisting of management modules, servermodules, and switching modules.
 12. The method in claim 1, wherein saidplurality of logical devices are configured as a logical manager. 13.The method in claim 1, wherein said plurality of logical devices areconfigured as a logical switcher.
 14. The method in claim 1, whereinsaid plurality of logical devices are configured as a logical router.15. The method in claim 1, wherein said plurality of logical devices areconfigured as a logical server.
 16. The method in claim 1, wherein saidplurality of logical devices comprise server modules.
 17. The method inclaim 1, wherein said plurality of logical devices comprise servermodules and said task is selected from the group of tasks consisting ofweb serving, streaming media server, caching server, file serving,application serving, and combinations thereof.
 18. The method in claim1, wherein said plurality of physical devices include a plurality ofdata storage devices, and said coordination and control of said physicaldevices into said logical device comprise managing said storage devicesas a redundant array of independent disks (RAID).
 19. The method inclaim 18, wherein said plurality of data storage devices comprise aplurality of disk drives managed as a mirroring data storage subsystem.20. The method in claim 18, wherein said RAID is managed as either aRAID Level 1 or RAID Level 0+1.
 21. The method in claim 18, wherein saidplurality of storage devices comprise hard disk drives and operation ofsaid hard disk drives is managed to reduce power consumption and prolongdisk drive life.
 22. The method in claim 19, wherein said managingincludes: selecting a first set of disc drives from among a plurality ofsets of disc drives as a currently active set; selecting a second set ofdisc drives as a currently inactive set and placing said selectedinactive set in a power conserving mode; using said first set of discdrives to retrieve data in response to read data requests; activatingsaid second set of disc drives in response to a write data request priorto performing said requested write operation; performing said writeoperation to both said first and second set of disc drives; andselecting one of said first set and second set as the inactive set anddeactivating said selected inactive set after performing said writeoperation.
 23. The method in claim 22, wherein said disc drives areconfigured as a RAID Level
 1. 24. The method in claim 22, wherein saiddisc drives are configured as a RAID Level 0+1.
 25. The method in claim22, wherein said disc drives are configured as mirrored storage each setduplicating the data of the other set.
 26. The method in claim 1,wherein said plurality of grouped network devices are electricallycoupled via a backplane bus and the logical grouping of the plurality ofnetwork devices into a single logical network device is performed undercontrol of software.
 27. The method in claim 1, wherein said softwareexecutes within a processor and memory associated within each networkdevice.
 28. The method in claim 1, wherein a plurality of networkdevices of the same type are grouped together and treated as a singlenetwork device.
 29. The method in claim 1, wherein said group of networkdevices treated as a single network device is managed and controlled asa single network device.
 30. The method in claim 1, wherein said groupof network devices treated as a single network device is power managedas a single network device.
 31. The method in claim 1, wherein saidgroup of network devices treated as a single network device is monitoredas a single network device.
 32. The method in claim 1, wherein saidplurality of grouped network devices are electrically coupled via abackplane bus and the logical grouping of the plurality of networkdevices into a single logical network device is performed under controlof software.
 33. The method in claim 1, wherein said software executeswithin a processor and memory associated within each network device. 34.The method in claim 1, wherein said plurality of network devices eachcomprise a server group.
 35. The method in claim 34, wherein each saidserver group comprises a plurality of server modules.
 36. The method inclaim 1, wherein said plurality of network devices each comprise acomputer node.
 37. The method in claim 36, wherein each computer node isconfigured as a computer node selected from the group consisting of aweb server, a streaming media server, a cache server, a file server, anapplication server, and a router.
 38. The method in claim 1, whereinsaid logical device is managed as a single image.
 39. The method inclaim 1, wherein said method includes a first plurality of physicaldevices grouped as a first logical device, and a second plurality ofphysical devices grouped as a second logical devices.
 40. The method inclaim 39, wherein said first and second logical devices are powermanaged as logical devices.
 41. The method in claim 1, wherein tasksperformed by each of the logical devices are monitored for each logicaldevice, for each physical device within said logical device, or both.42. The method in claim 1, wherein demand for each of a plurality oftasks performed by the logical device are monitored for the logicaldevice, for each physical device within said logical device, or both;and said task demand information is used to power manage said logicaldevice and said physical devices.
 43. The method in claim 42, whereinsaid power management includes reconfiguring said logical unit toprovide a required performance level with a desired power consumption.44. The method in claim 42, wherein said power management includesreconfiguring said logical unit to minimize power consumption whilemaintaining a required task performance level.
 45. The method in claim44, wherein said reconfiguring said logical unit to minimize powerconsumption while maintaining a required task performance level isperformed in accordance with power management policies.
 46. The methodin claim 44, wherein said reconfiguring said logical unit to extend thelife of a component constituting said physical device.
 47. The method inclaim 45, wherein said power management policies include a policy thatattempts to satisfy performance requirements by operating each of firstselected physical devices at first performance levels, and a each ofsecond selected physical devices at second performance levels.
 48. Themethod in claim 47, wherein said second performance levels are aperformance level at which substantially no-tasks are performed andpower consumption is reduced relative to said first performance levels.49. The method in claim 48, wherein said first performance levels arebetween about 40 percent and about 100 percent of maximum performancelevel.
 50. The method in claim 45, wherein said power managementpolicies include a policy that attempts to satisfy performancerequirements by operating a minimum number of physical devices tosatisfy task performance requirements.
 51. The method in claim 40,wherein performance of a logical group of physical network devicesforming a single logical device are power managed by reducing theperformance and power consumption of each constituent physical device inpredetermined equal increments, predetermined unequal increments, in asubstantially continuous manner, or dynamically in a continuous orincremental manner.
 52. The method in claim 51, wherein said unequalincrements include placing one or more of said plurality of physicaldevices in an operating mode wherein said physical device includes aprocessor operating to perform tasks only when a processor clock signalis provided, and said processor clock signal is stopped or substantiallystopped.
 53. The method in claim 52, wherein said processor is placed ina suspend mode.
 54. The method in claim 51, wherein said unequalincrements include placing one or more of said plurality of physicaldevices in a sleep mode.
 55. The method in claim 51, wherein saidunequal increments include placing one or more of said plurality ofphysical devices in a powered-off mode.
 56. The method in claim 45,wherein a composite performance of a logical network device is achievedby placing some physical network devices in a first power saving modeand by placing others in a different mode.
 57. A method of reducingpower consumption in a computer system having a mirroring data storagesubsystem, said method comprising steps of: selecting a first set ofdisc drives from among a plurality of sets of disc drives as a currentlyactive set; selecting a second set of disc drives as a currentlyinactive set and placing said selected inactive set in a powerconserving mode; using said first set of disc drives to retrieve data inresponse to read data requests; activating said second set of discdrives in response to a write data request prior to performing saidrequested write operation; performing said write operation to both saidfirst and second set of disc drives; and selecting one of said first setand second set as the inactive set and deactivating said selectedinactive set after performing said write operation.
 58. The method inclaim 57, wherein said disc drives are configured as a RAID Level
 1. 59.The method in claim 57, wherein said disc drives are configured as aRAID Level 0+1.
 60. The method in claim 57, wherein said disc drives areconfigured as mirrored storage each set duplicating the data of theother set.
 61. The method in claim 57, wherein: said disc drives aredisposed on separate server modules that have been grouped as a singlelogical device; and said disk drives are configured as either a RAIDLevel 1 or a RAID Level 0+1 and include data mirroring.
 62. A methodaccording to claim 1, wherein the act of determining demand for the taskincludes determining a frequency of requests received for the task. 63.A method according to claim 1, wherein the act of determining demand forthe task includes determining a quality of service requirement for thetask.
 64. A method according to claim 1, wherein the act of increasingor decreasing the number of physical network devices configured toperform the task includes sending a signal to at least one of thenetwork devices to enter a power saving mode.
 65. A method according toclaim 1, wherein the act of increasing or decreasing the number ofphysical network devices configured to perform the task includesreprovisioning at least one of the plurality of physical network devicesto perform the task.